Miracle-dz/UniFAR-FTUs huggingface dataset:
PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models

The size of Transformer models is growing at an unprecedented rate. It has taken less than one year to reach trillion-level parameters since the release of GPT-3 (175B). Training such models requires both substantial engineering efforts and enormous computing resources, which are luxu-ries most research teams cannot afford. In this paper, we propose PipeTransformer , which leverages automated elastic pipelining for efﬁ-cient distributed training of Transformer models. In PipeTransformer , we design an adaptive on the ﬂy freeze algorithm that can identify and freeze some layers gradually during training, and an elastic pipelining system that can dynamically allocate resources to train the remaining active layers. More speciﬁcally, PipeTransformer automatically excludes frozen layers from the pipeline, packs active layers into fewer GPUs, and forks more replicas to increase data-parallel width. We evaluate PipeTransformer us-ing Vision Transformer (ViT) on ImageNet and BERT on SQuAD and GLUE datasets. Our results show that compared to the state-of-the-art base-line, PipeTransformer attains up to 2 . 83 - fold speedup without losing accuracy. We also provide various performance analyses for a more comprehensive understanding of our algorithmic and system-wise design. Finally, we have modularized our training system with ﬂexible APIs and made the source code publicly available at https://DistML.ai .
Provably Efficient Fictitious Play Policy Optimization for Zero-Sum Markov Games with Structured Transitions

While single-agent policy optimization in a fixed environment has attracted a lot of research attention recently in the reinforcement learning community, much less is known theoretically when there are multiple agents playing in a potentially competitive environment. We take steps forward by proposing and analyzing new fictitious play policy optimization algorithms for zero-sum Markov games with structured but unknown transitions. We consider two classes of transition structures: factored independent transition and single-controller transition. For both scenarios, we prove tight $\widetilde{\mathcal{O}}(\sqrt{K})$ regret bounds after $K$ episodes in a two-agent competitive game scenario. The regret of each agent is measured against a potentially adversarial opponent who can choose a single best policy in hindsight after observing the full policy sequence. Our algorithms feature a combination of Upper Confidence Bound (UCB)-type optimism and fictitious play under the scope of simultaneous policy optimization in a non-stationary environment. When both players adopt the proposed algorithms, their overall optimality gap is $\widetilde{\mathcal{O}}(\sqrt{K})$.
LAMDA: Label Matching Deep Domain Adaptation

Deep domain adaptation (DDA) approaches have recently been shown to perform better than their shallow rivals with better modeling capacity on complex domains (e.g., image, structural data, and sequential data). The underlying idea is to learn domain invariant representations on a latent space that can bridge the gap between source and target domains. Several theoretical studies have established insightful understanding and the beneﬁt of learning domain invariant features; however, they are usually limited to the case where there is no label shift, hence hindering its applicability. In this paper, we propose and study a new challenging setting that allows us to use a Wasserstein distance (WS) to not only quantify the data shift but also to deﬁne the label shift directly. We further develop a theory to demonstrate that minimizing the WS of the data shift leads to closing the gap between the source and target data distributions on the latent space (e.g., an intermediate layer of a deep net), while still being able to quantify the label shift with respect to this latent space. Interestingly, our theory can consequently explain certain drawbacks of learning domain invariant features on the latent space. Finally, grounded on the results and guidance of our developed theory, we propose the Label Matching Deep Domain Adaptation (LAMDA) approach that outperforms baselines on real-world datasets for DA problems.
Bayesian Deep Learning via Subnetwork Inference

The Bayesian paradigm has the potential to solve core issues of deep neural networks such as poor calibration and data inefficiency. Alas, scaling Bayesian inference to large weight spaces often requires restrictive approximations. In this work, we show that it suffices to perform inference over a small subset of model weights in order to obtain accurate predictive posteriors. The other weights are kept as point estimates. This subnetwork inference framework enables us to use expressive, otherwise intractable, posterior approximations over such subsets. In particular, we implement subnetwork linearized Laplace as a simple, scalable Bayesian deep learning method: We first obtain a MAP estimate of all weights and then infer a full-covariance Gaussian posterior over a subnetwork using the linearized Laplace approximation. We propose a subnetwork selection strategy that aims to maximally preserve the model's predictive uncertainty. Empirically, our approach compares favorably to ensembles and less expressive posterior approximations over full networks. Our proposed subnetwork (linearized) Laplace method is implemented within the laplace PyTorch library at https://github.com/AlexImmer/Laplace.
Narrow Margins: Classification, Margins and Fat Tails

It is well-known that, for separable data, the reg-ularised two-class logistic regression or support vector machine re-normalised estimate converges to the maximal margin classiﬁer as the regular-isation hyper-parameter λ goes to 0. The fact that different loss functions may lead to the same solution is of theoretical and practical relevance as margin maximisation allows more straightforward considerations in terms of generalisation and geometric interpretation. We investigate the case where this convergence property is not guaranteed to hold and show that it can be fully characterised by the distribution of error terms in the latent variable interpretation of linear classiﬁers. In particular, if errors follow a regularly varying distribution, then the regularised and re-normalised estimate does not converge to the maximal margin classiﬁer. This shows that classiﬁcation with fat tails has a qualitatively different behaviour, which should be taken into account when considering real-life data.
Learner-Private Convex Optimization

Convex optimization with feedback is a framework where a learner relies on iterative queries and feedback to arrive at the minimizer of a convex function. It has gained considerable popularity thanks to its scalability in large-scale optimization and machine learning. The repeated interactions, however, expose the learner to privacy risks from eavesdropping adversaries that observe the submitted queries. In this paper, we study how to optimally obfuscate the learner’s queries in convex optimization with first-order feedback, so that their learned optimal value is provably difficult to estimate for an eavesdropping adversary. We consider two formulations of learner privacy: a Bayesian formulation in which the convex function is drawn randomly, and a maximin formulation in which the function is fixed and the adversary’s probability of error is measured with respect to a minimax criterion. Suppose that the learner wishes to ensure the adversary cannot estimate accurately with probability greater than <inline-formula> <tex-math notation="LaTeX">$1/L$ </tex-math></inline-formula> for some <inline-formula> <tex-math notation="LaTeX">$L > 0$ </tex-math></inline-formula>. Our main results show that the query complexity overhead is additive in <inline-formula> <tex-math notation="LaTeX">$L$ </tex-math></inline-formula> in the maximin formulation, but multiplicative in <inline-formula> <tex-math notation="LaTeX">$L$ </tex-math></inline-formula> in the Bayesian formulation. Compared to existing learner-private sequential learning models with binary feedback, our results apply to the significantly richer family of general convex functions with full-gradient feedback. Our proofs rely on tools from the theory of Dirichlet processes, as well as a novel strategy designed for measuring information leakage under a full-gradient oracle.
Aperiodic Sampled-Data H∞ Filtering for Lipschitz Nonlinear Systems: An Impulsive System Approach

This paper is concerned with the <inline-formula> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> filtering for Lipschitz nonlinear systems under aperiodically sampled measurements. The developed filter is a hybrid system, whose states undergo a change or reset at sampling instants. The resulting filtering error system is then modelled as a kind of nonlinear impulsive systems. By introducing a time-varying Lyapunov functional candidate, a sufficient condition for the existence of desired filter is derived to ensure the filtering error system asymptotic stability and guarantee an <inline-formula> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> performance. The optimal <inline-formula> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> performance and corresponding filter gain matrix can be obtained by solving a convex problem with linear matrix inequalities (LMIs) constrains. Two examples are given to show the effectiveness of the theoretical results, and our results are less conservative than existing ones through comparisons.
The prevalence and genotype distribution of human papilloma virus in cervical squamous intraepithelial lesion and squamous cell carcinoma in Taizhou, China

Abstract Human papillomavirus (HPV) infection is a common sexually transmitted disease worldwide and the leading cause of cervical cancer. Current vaccines do not cover all HPV genotypes whereas the distribution of HPV genotypes varies in different geographic regions. The study aimed to investigate the distribution of HPV genotypes in patients with cervical squamous intraepithelial lesion (SIL) and cervical squamous cell carcinoma (SCC) in Taizhou City of Jiangsu Province, China. A total of 940 patients including 489 cases with cervical low-grade squamous intraepithelial lesions (LSIL), 356 cases with cervical high-grade squamous intraepithelial lesions (HSIL), and 95 cases with cervical SCC, underwent a biopsy or surgery in Taizhou People's Hospital between January 2019 and December 2019. The HPV testing results were retrospectively analyzed. The overall prevalence of any, high-risk, and low-risk HPV was 83.83%, 81.91%, and 12.13%, respectively. The 5 most common HPV genotypes were HPV16 (35.64%), HPV52 (16.91%), HPV58 (13.94%), HPV33 (8.94%), and HPV18 (7.98%). The prevalence of any and HR-HPV in SCC was significantly higher than those in LSIL and HSIL, while the prevalence of LR-HPV in SCC was significantly lower than those in LSIL and HSIL (P < .01). Single and dual HPV infections were prevalent in SCC, LSIL, and HSIL. Furthermore, the prevalence of dual HPV infection in SCC was significantly higher than those in LSIL and HSIL (P = .002). The HPV prevalence varied by age, being highest among women with SCC, LSIL, and HSIL aged 40 to 49 years, 40 to 49 years, and 50 to 59 years, respectively. In conclusion, the findings revealed a very high prevalence of HPV in women with cervical lesions in Taizhou. Routine HPV tests must cover all common HPV genotypes in clinical practice.
5,6-Dihydroxyindole-2-carboxylic Acid–TiO2 Charge Transfer Complexes in the Radical Polymerization of Melanogenic Precursor(s)

A combination of biomedical and technological applications is generating, over the past decades, the well-established interest toward melanins and melanogenesis. Several compounds have been explored to promote/catalyze oxidative polymerization of melanogenic precursors, such as 5,6-dihydroxyindole-2-carboxylic acid (DHICA), to melanin-like biopolymers in vitro. TiO2 has shown a photocatalytic activity driving DHICA polymerization and leading to the formation of melanin–TiO2 hybrid nanostructures with unique biocide behavior even under visible light. However, the mechanism of melanin formation in those hybrids is not yet well understood although a ligand to metal charge transfer (LMCT) process involving DHICA and Ti4+ ions was hypothesized. Here, we focus on melanin formation and apply a complementary analysis, by using photoluminescence (PL), UV–vis, electron paramagnetic resonance (EPR), and nuclear magnetic resonance (NMR) spectroscopy to reveal the mechanism of DHICA polymerization in the presence of a.
The relationship between school administrators’ transformational leadership behaviours and teachers’ perceptions of organizational justice

The aim of this study is to determine the relationship between school administrators’ transformational leadership behaviours and teachers’ perceptions of organizational justice. The sample of the study consists of 170 teachers working in high schools in Elazığ city center. In the study, the Transformational Leadership Scale was used to determine whether school administrators exhibit transformational leadership behaviours, and the Organizational Justice Scale was used to determine teachers' perceptions of organizational justice. Pearson moment two-correlation analysis and multiple regression analysis techniques were used in the analysis of the data. According to the research findings, it was observed that school administrators frequently exhibited transformational leadership behaviours and teachers' perceptions of organizational justice were at the level of “I agree”. In general, moderate, positive and significant relationships were found between transformational leadership and organizational justice. According to the results of the regression analysis, it was determined that the idealized influence dimension of transformational leadership positively and significantly predicted the formal procedures dimension of the organizational justice scale, and also idealized influence and individualized consideration dimensions of transformational leadership positively and significantly predicted the interactional justice dimension of the organizational justice scale. The research results were discussed in the relevant literature and recommendations were made.
Cellulose Nanofibers from Olive Tree Pruning as Food Packaging Additive of a Biodegradable Film

A biodegradable packaging film containing cellulose nanofibers from olive tree pruning, a by-product of olives production, was obtained using a solvent casting method. Nanocellulose was added to polyvinyl alcohol (PVA) to enhance the technological properties of the composite film as food packaging material. Nanocellulose was obtained from unbleached and bleached pulp through a mechanical and TEMPO pretreatment. Crystalline and chemical structure, surface microstructure, UV and gas barrier, optical, mechanical and antioxidant properties, as well as thermal stability were evaluated. Regarding optical properties, the UV barrier was increased from 6% for the pure PVA film to 50% and 24% for unbleached and bleached nanocellulose, respectively. The antioxidant capacity increased significantly in unbleached mechanical nanocellulose-films (5.3%) compared to pure PVA film (1.7%). In terms of mechanical properties, the tensile strength of the 5% unbleached mechanical nanocellulose films was significantly improved compared to the pure PVA film. Similarly, the 5% nanocellulose films had increased the thermal stability and improved barrier properties, reducing water vapor permeability by 38–59% and presenting an oxygen barrier comparable to aluminum layer and plastic films. Our results support the use of the developed films as a green alternative material for food packaging.
Ergenlerde Sürekli Öfke ve Öfke İfade Tarzlarının Çeşitli Değişkenler Açısından İncelenmesi

Sürekli Öfke, Öfke İfade Tarzı, Ergenler. Bu araştırmanın amacı, ergenlerin sürekli öfke ve öfke ifade tarzlarının çeşitli değişkenler açısından incelenmesidir. Araştırmanın çalışma grubunu Orta Anadolu’da bir ilde 5 farklı ortaöğretim kurumuna devam eden 534 lise öğrencisi oluşturmaktadır. Katılımcılar 250 (%46.8) erkek, 284 (%53.2) kadından oluşmaktadır. Araştırmada ergenlerin sürekli öfke ve öfke ifade tarzlarını belirlemek için Sürekli Öfke ve Öfke İfade Tarz Ölçeği ve Kişisel Bilgi Formu kullanılmıştır. Verilerin analizinde; ttesti ve tek yönlü varyans analizi kullanılmıştır. Araştırma bulgularına göre, ergenlerin cinsiyet değişkenine göre sürekli öfke ve öfke ifade tarzı alt ölçeklerinden, sürekli öfke ve öfke dışa alt boyutlarında kadın öğrenciler lehine anlamlı farklılık bulunmuştur. Öfke kontrol alt boyutunda ise erkek öğrenciler lehine anlamlı farklılık bulunmuştur. Sınıf değişkenine göre sürekli öfke ve öfke ifade tarzı alt ölçeklerinden öfke içte alt boyutu 10. sınıf öğrencilerinde 9. sınıf öğrencilerine göre anlamlı düzeyde yüksek bulunmuştur. Anne ve baba eğitim düzeyi değişkenine göre sürekli öfke ve öfke ifade tarz alt ölçeklerinden alınan puanlar anlamlı farklılık bulunmamıştır. Öfke, ergenlik döneminde hissedilen en yoğun duygulardan bir tanesidir. Bu çalışmanın bulguları ilgili literatür desteği ile tartışılmıştır.
Corbino FETs in magnetic ﬁeld: highly tunable photodetectors

We study gated ﬁeld eﬀect transistors (FETs) with an eccentric Corbino-disk geometry, such that the drain spans its circumference while the oﬀ-center inner ring acts as a source. An AC THz potential diﬀerence is applied between source and gate while a static source-drain voltage, rectiﬁed by the nonlinearities of FET electrons, is measured. When a magnetic ﬁeld is applied perpendicular to the device, a strong resonance appears at the cyclotron frequency. The strength of the resonance can be tuned by changing the eccentricity of the disk. We show that there is an optimum value of the eccentricity that maximizes the responsivity of the FET.
Diffusivity Measurement by Single-Molecule Recycling in a Capillary Microchannel

Microfluidic devices have been extensively investigated in recent years in fields including ligand-binding analysis, chromatographic separation, molecular dynamics, and DNA sequencing. To prolong the observation of a single molecule in aqueous buffer, the solution in a sub-micron scale channel is driven by a electric field and reversed after a fixed delay following each passage, so that the molecule passes back and forth through the laser focus and the time before irreversible photobleaching is extended. However, this practice requires complex chemical treatment to the inner surface of the channel to prevent unexpected sticking to the surface and the confined space renders features, such as a higher viscosity and lower dielectric constant, which slow the Brownian motion of the molecule compared to the bulk solution. Additionally, electron beam lithography used for the fabrication of the nanochannel substantially increases the cost, and the sub-micron dimensions make the molecule difficult to locate. In this paper, we propose a method of single-molecule recycling in a capillary microchannel. A commercial fused-silica capillary with an inner diameter of 2 microns is chopped into a 1-inch piece and is fixed onto a cover slip. Two o-rings on the sides used as reservoirs and an o-ring in the middle used as observation window are glued over the capillary. The inner surface of the capillary is chemically processed to reduce the non-specific sticking and to improve capillary effect. The device does not require high-precision fabrication and thus is less costly and easier to prepare than the nanochannel. 40 nm Fluospheres® in 50% methanol are used as working solution. The capillary is translated by a piezo stage to recycle the molecule, which diffuses freely through the capillary, and a confocal microscope is used for fluorescence collection. The passing times of the molecule through the laser focus are calculated by a real-time control system based on an FPGA, and the commands of translation are given to the piezo stage through a feedback algorithm. The larger dimensions of the capillary overcomes the strong sticking, the reduced diffusivity, and the difficulty of localizing the molecule. We have achieved a maximum number of recycles of more than 200 and developed a maximum-likelihood estimation of the diffusivity of the molecule, which attains results of the same magnitude as the previous report. This technique simplifies the overall procedure of the single-molecule recycling and could be useful for the ligand-binding studies in high-throughput screening.
Artifacts in eld free line magnetic particle imaging in the presence of inhomogeneous and nonlinear magnetic elds

Introduction: Magnetic Particle Imaging (MPI) is an emergingmedical imagingmodality that detects superparamagnetic particles exploiting their nonlinear magnetization response. Spatial encoding can be realized using a Field Free Line (FFL), which is generated, rotated and translated through the Field of View (FOV) using a combination of magnetic gradient elds and homogeneous excitation elds. When scaling up systems and/or enlarging the FOV in comparison to the scanner bore, ensuring homogeneity and linearity of the magnetic elds becomes challenging. The present contribution describes the rst comprehensive, systematic study on the in uence of magnetic eld imperfections in FFL MPI. Methods: In a simulation study, 14 di erent FFL scanner setups have been examined. Starting from an ideal scanner using perfect magnetic elds, de ned imperfections have been introduced in a range of con gurations (nonlinear gradient elds, inhomogeneous excitation elds, or inhomogeneous receive elds, or a combination thereof). In the rst part of the study, the voltage induced in the receive channels parallel and perpendicular to the FFL translation have been studied for discrete FFL angles. In the second part, an imaging process has been simulated comparing di erent image reconstruction approaches. Results: The induced voltage signals demonstrate illustratively the e ect of themagnetic eld imperfections. In images reconstructed using a Radon-based approach, the magnetic eld imperfections lead to pronounced artifacts, especially if a deconvolution using the point spread function is performed. In images reconstructed using a system function based approach, variations in local image quality become visible. Conclusion: For Radon-based image reconstruction in FFLMPI in *Corresponding Author: Hanne Medimagh: Institute of Medical Engineering, University of Luebeck, Ratzeburger Allee 160, Luebeck, Germany, E-mail: medimagh@imt.uni-luebeck.de Patrick Weissert, Gael Bringout, Klaas Bente, Matthias Weber, Ksenija Gräfe, Aileen Cordes, Thorsten M. Buzug: Institute of Medical Engineering, University of Luebeck, Ratzeburger Allee 160, Luebeck, Germany, E-mail: weissert@imt.uni-luebeck.de, bringout@imt.uni-luebeck.de, bente@imt.uni-luebeck.de, weber@imt.uni-luebeck.de, graefe@imt.uni-luebeck.de, cordes@imt.uni-luebeck.de, buzug@imt.uni-luebeck.de the presence of inhomogeneous and nonlinear magnetic elds, artifact correction methods will have to be developed. In this regard, a rst approachhas recently beenpresented by another group. Further research is required to elucidate the in uence of magnetic eld imperfections in MPI using a system function based approach.