Miracle-dz/UniFAR-FTUs huggingface dataset:
Protein Representation Learning by Geometric Structure Pretraining

Learning effective protein representations is critical in a variety of tasks in biology such as predicting protein function or structure. Existing approaches usually pretrain protein language models on a large number of unlabeled amino acid sequences and then finetune the models with some labeled data in downstream tasks. Despite the effectiveness of sequence-based approaches, the power of pretraining on known protein structures, which are available in smaller numbers only, has not been explored for protein property prediction, though protein structures are known to be determinants of protein function. In this paper, we propose to pretrain protein representations according to their 3D structures. We first present a simple yet effective encoder to learn the geometric features of a protein. We pretrain the protein graph encoder by leveraging multiview contrastive learning and different self-prediction tasks. Experimental results on both function prediction and fold classification tasks show that our proposed pretraining methods outperform or are on par with the state-of-the-art sequence-based methods, while using much less pretraining data. Our implementation is available at https://github.com/DeepGraphLearning/GearNet.
MIMICause: Representation and automatic extraction of causal relation types from clinical notes

Understanding causal narratives communicated in clinical notes can help make strides towards personalized healthcare. Extracted causal information from clinical notes can be combined with structured EHR data such as patients’ demographics, diagnoses, and medications. This will enhance healthcare providers’ ability to identify aspects of a patient’s story communicated in the clinical notes and help make more informed decisions. In this work, we propose annotation guidelines, develop an annotated corpus and provide baseline scores to identify types and direction of causal relations between a pair of biomedical concepts in clinical notes; communicated implicitly or explicitly, identified either in a single sentence or across multiple sentences. We annotate a total of 2714 de-identified examples sampled from the 2018 n2c2 shared task dataset and train four different language model based architectures. Annotation based on our guidelines achieved a high inter-annotator agreement i.e. Fleiss’ kappa (\kappa) score of 0.72, and our model for identification of causal relations achieved a macro F1 score of 0.56 on the test data. The high inter-annotator agreement for clinical text shows the quality of our annotation guidelines while the provided baseline F1 score sets the direction for future research towards understanding narratives in clinical texts.
PillarGrid: Deep Learning-Based Cooperative Perception for 3D Object Detection from Onboard-Roadside LiDAR

3D object detection plays a fundamental role in enabling driving automation, which is regarded as a significant leap forward for contemporary transportation systems from the perspectives of safety, mobility, and sustainability. Most of the state-of-the-art object detection methods from point clouds are developed based on a single onboard LiDAR, whose performance will be inevitably limited by the range and occlusion, especially in dense traffic scenarios. In this paper, we propose PillarGrid, a novel cooperative perception method fusing information from multiple 3D LiDARs (both on-board and roadside), to enhance the situation awareness for connected and automated vehicles (CAVs). PillarGrid consists of four main components: 1) cooperative preprocessing of point clouds, 2) pillar-wise voxelization and feature extraction, 3) grid-wise deep fusion of features from multiple sensors, and 4) convolutional neural network (CNN)-based augmented 3D object detection. A novel cooperative perception platform is developed for model training and testing. Extensive experimentation shows that PillarGrid outperforms other single-LiDAR-based 3D object detection methods concerning both accuracy and range by a large margin.
ORDSIM: Ordinal Regression for E-Commerce Query Similarity Prediction

Query similarity prediction task is generally solved by regression based models with square loss. Such a model is agnostic of absolute similarity values and it penalizes the regression error at all ranges of similarity values at the same scale. However, to boost e-commerce platform's monetization, it is important to predict high-level similarity more accurately than low-level similarity, as highly similar queries retrieves items according to user-intents, whereas moderately similar item retrieves related items, which may not lead to a purchase. Regression models fail to customize its loss function to concentrate around the high-similarity band, resulting poor performance in query similarity prediction task. We address the above challenge by considering the query prediction as an ordinal regression problem, and thereby propose a model, ORDSIM (ORDinal Regression for SIMilarity Prediction). ORDSIM exploits variable-width buckets to model ordinal loss, which penalizes errors in high-level similarity harshly, and thus enable the regression model to obtain better prediction results for high similarity values. We evaluate ORDSIM on a dataset of over 10 millions e-commerce queries from eBay platform and show that ORDSIM achieves substantially smaller prediction error compared to the competing regression methods on this dataset.
Categorization model of moving small-scale intensity enhancements in solar active regions

The small-scale moving intensity enhancements remotely observed in the extreme ultraviolet images of the solar active regions, which we refer to as active region moving campfires (ARMCs), are related to local plasma temperature and/or density enhancements. Their dynamics is driven by the physical processes in the entire coronal plasma. Our previous study of ARMCs indicates that they have characteristic velocities at around the background sound speed. The main goal of our work is to carry out a simultaneous analysis of EUV images from two observational missions, SDO/AIA and Hi-C 2.1. The aims of the performed cross-validating analysis of both SDO/AIA and Hi-C 2.1 data were to reveal how the observed moving features are distributed over the studied active region, AR12712, test the existence of different groups of ARMCs with distinct physical characteristics. We use the statistical model of intensity centroid convergence and tracking that was developed in our previous paper. Furthermore, a Gaussian mixture model fit of the observed complex of moving ARMCs is elaborated to reveal the existence of distinct ARMC groups and to study the physical characteristics of these different groups. In data from the 171\AA, 193\AA\ and 211\AA\ channels of SDO/AIA, we identified several groups of ARMCs with respect to both blob intensity and velocity profiles. The existence of such groups is confirmed by the cross-validation of the 172\AA\ data sets from Hi-C 2.1. The ARMCs studied in this paper have characteristic velocities in the range of the typical sound speeds in coronal loops. Hence, these moving objects differ from the well-known rapid Alfv\'enic velocity jets from magnetic reconnection sites. This is also proven by the fact that ARMCs propagate along the active region magnetic structure (strands). The nature of the discovered statistical grouping of the ARMC events is not known.
Semiparametric doubly robust targeted double machine learning: a review

In this review we cover the basics of efficient nonparametric parameter estimation (also called functional estimation), with a focus on parameters that arise in causal inference problems. We review both efficiency bounds (i.e., what is the best possible performance for estimating a given parameter? ) and the analysis of particular estimators (i.e., what is this estimator's error, and does it attain the efficiency bound? ) under weak assumptions. We emphasize minimax-style efficiency bounds, worked examples, and practical shortcuts for easing derivations. We gloss over most technical details, in the interest of highlighting important concepts and providing intuition for main ideas.
Measuring the CP properties of the Higgs sector at electron-positron colliders

The violation of CP symmetry at the electro-weak scale is one of the essential ingredients for electro-weak baryogenesis. It is therefore of great interest to map the CP properties of the Higgs sector in as much detail as possible. A linear electron-positron Higgs factory collider will provide many opportunities to probe the CP nature of the Higgs sector, thanks to access to several Higgs production processes at a wide range of centre-of-mass energies. In this paper we report on two studies based on full simulation of detector response and realistic reconstruction algorithms: 1) a study of $h \to \tau \tau$ at ILC-250, in which mixing of CP eigenstates can be measured to a precision of 75~mrad; and 2) the current status of an ongoing study of the $ZZ$--fusion process at 1.4 TeV CLIC and 1 TeV ILC, for which we expect to achieve concrete results during the Snowmass study period.
A p-adic Descartes solver: the Strassman solver

Solving polynomials is a fundamental computational problem in mathematics. In the real setting, we can use Descartes' rule of signs to efficiently isolate the real roots of a square-free real polynomial. In this paper, we translate this method into the $p$-adic worlds. We show how the $p$-adic analog of Descartes' rule of signs, Strassman's theorem, leads to an algorithm to isolate the roots of a square-free $p$-adic polynomial. Moreover, we show that this algorithm runs in $\mathcal{O}(d^2\log^3d)$-time for a random $p$-adic polynomial of degree $d$. To perform this analysis, we introduce the condition-based complexity framework from real/complex numerical algebraic geometry into $p$-adic numerical algebraic geometry.
Dark Matter Physics from the CMB-S4 Experiment

The nature of dark matter is one of the major puzzles of fundamental physics, integral to the understanding of our universe across almost every epoch. The search for dark matter takes place at different energy scales, and use data ranging from particle colliders to astrophysical surveys. We focus here on CMB-S4, a future ground-based Cosmic Microwave Background (CMB) experiment, which is expected to provide exquisite measurements of the CMB temperature and polarization anisotropies. These measurements (on their own and in combination with other surveys) will allow for new means to shed light on the nature of dark matter.
Accelerating Plug-and-Play Image Reconstruction via Multi-Stage Sketched Gradients

In this work we propose a new paradigm for designing fast plug-and-play (PnP) algorithms using dimensionality reduction techniques. Unlike existing approaches which utilize stochastic gradient iterations for acceleration, we propose novel multi-stage sketched gradient iterations which first perform downsampling dimensionality reduction in the image space, and then efficiently approximate the true gradient using the sketched gradient in the low-dimensional space. This sketched gradient scheme can also be naturally combined with PnP-SGD methods for further improvement on computational complexity. As a generic acceleration scheme, it can be applied to accelerate any existing PnP/RED algorithm. Our numerical experiments on X-ray fan-beam CT demonstrate the remarkable effectiveness of our scheme, that a computational free-lunch can be obtained using this dimensionality reduction in the image space.
A Comparative Study on Forecasting of Retail Sales

Predicting product sales of large retail companies is a challenging task considering volatile nature of trends, seasonalities, events as well as unknown factors such as market competitions, change in customer's preferences, or unforeseen events, e.g., COVID-19 outbreak. In this paper, we benchmark forecasting models on historical sales data from Walmart to predict their future sales. We provide a comprehensive theoretical overview and analysis of the state-of-the-art timeseries forecasting models. Then, we apply these models on the forecasting challenge dataset (M5 forecasting by Kaggle). Specifically, we use a traditional model, namely, ARIMA (Autoregressive Integrated Moving Average), and recently developed advanced models e.g., Prophet model developed by Facebook, light gradient boosting machine (LightGBM) model developed by Microsoft and benchmark their performances. Results suggest that ARIMA model outperforms the Facebook Prophet and LightGBM model while the LightGBM model achieves huge computational gain for the large dataset with negligible compromise in the prediction accuracy.
Improving Macroeconomic Model Validity and Forecasting Performance with Pooled Country Data using Structural, Reduced Form, and Neural Network Model

We show that pooling countries across a panel dimension to macroeconomic data can improve by a statistically significant margin the generalization ability of structural, reduced form, and machine learning (ML) methods to produce state-of-the-art results. Using GDP forecasts evaluated on an out-of-sample test set, this procedure reduces root mean squared error by 12% across horizons and models for certain reducedform models and by 24% across horizons for dynamic structural general equilibrium models. Removing US data from the training set and forecasting out-of-sample country-wise, we show that reduced-form and structural models are more policy-invariant when trained on pooled data, and outperform a baseline that uses US data only. Given the comparative advantage of ML models in a data-rich regime, we demonstrate that our recurrent neural network model and automated ML approach outperform all tested baseline economic models. Robustness checks indicate that our outperformance is reproducible, numerically stable, and generalizable across models.
Federated Cycling (FedCy): Semi-Supervised Federated Learning of Surgical Phases

Recent advancements in deep learning methods bring computer-assistance a step closer to fulfilling promises of safer surgical procedures. However, the generalizability of such methods is often dependent on training on diverse datasets from multiple medical institutions, which is a restrictive requirement considering the sensitive nature of medical data. Recently proposed collaborative learning methods such as Federated Learning (FL) allow for training on remote datasets without the need to explicitly share data. Even so, data annotation still represents a bottleneck, particularly in medicine and surgery where clinical expertise is often required. With these constraints in mind, we propose FedCy, a federated semi-supervised learning (FSSL) method that combines FL and self-supervised learning to exploit a decentralized dataset of both labeled and unlabeled videos, thereby improving performance on the task of surgical phase recognition. By leveraging temporal patterns in the labeled data, FedCy helps guide unsupervised training on unlabeled data towards learning task-specific features for phase recognition. We demonstrate significant performance gains over state-of-the-art FSSL methods on the task of automatic recognition of surgical phases using a newly collected multi-institutional dataset of laparoscopic cholecystectomy videos. Furthermore, we demonstrate that our approach also learns more generalizable features when tested on data from an unseen domain.
Symbolic Learning to Optimize: Towards Interpretability and Scalability

Recent studies on Learning to Optimize (L2O) suggest a promising path to automating and accelerating the optimization procedure for complicated tasks. Existing L2O models parameterize optimization rules by neural networks, and learn those numerical rules via meta-training. However, they face two common pitfalls: (1) scalability: the numerical rules represented by neural networks create extra memory overhead for applying L2O models, and limit their applicability to optimizing larger tasks; (2) interpretability: it is unclear what an L2O model has learned in its black-box optimization rule, nor is it straightforward to compare different L2O models in an explainable way. To avoid both pitfalls, this paper proves the concept that we can"kill two birds by one stone", by introducing the powerful tool of symbolic regression to L2O. In this paper, we establish a holistic symbolic representation and analysis framework for L2O, which yields a series of insights for learnable optimizers. Leveraging our findings, we further propose a lightweight L2O model that can be meta-trained on large-scale problems and outperformed human-designed and tuned optimizers. Our work is set to supply a brand-new perspective to L2O research. Codes are available at: https://github.com/VITA-Group/Symbolic-Learning-To-Optimize.
A Bayesian Precision Response-adaptive Phase II Clinical Trial Design for Radiotherapies with Competing Risk Survival Outcomes

Many phase II clinical trials have used survival outcomes as the primary endpoints in recent decades. Suppose the radiotherapy is evaluated in a phase II trial using survival outcomes. In that case, the competing risk issue often arises because the time to disease progression can be censored by the time to normal tissue complications, and vice versa. Besides, much literature has examined that patients receiving the same radiotherapy dose may yield distinct responses due to their heterogeneous radiation susceptibility statuses. Therefore, the “one-dose-fit-all” strategy often fails, and it is more relevant to evaluate the subgroup-specific treatment effect with the subgroup defined by the radiation susceptibility status. In this paper, we propose a Bayesian precision phase II trial design evaluating the subgroup-specific treatment effects of radiotherapy. We use the cause-specific hazard approach to model the competing risk survival outcomes. We propose restricting the candidate radiation doses based on each patient’s radiation susceptibility status. Only the clinically feasible personalized dose will be considered, which enhances the benefit for the patients in the trial. In addition, we propose a stratified Bayesian adaptive randomization scheme such that more patients will be randomized to the dose reporting more favorable survival outcomes. Numerical studies have shown that the proposed design performed well and outperformed the conventional design ignoring the competing risk issue.
Computing the action ground state for the rotating nonlinear Schrödinger equation

We consider the computations of the action ground state for a rotating nonlinear Schr\"odinger equation. It reads as a minimization of the action functional under the Nehari constraint. In the focusing case, we identify an equivalent formulation of the problem which simplifies the constraint. Based on it, we propose a normalized gradient flow method with asymptotic Lagrange multiplier and establish the energy-decaying property. Popular optimization methods are also applied to gain more efficiency. In the defocusing case, we prove that the ground state can be obtained by the unconstrained minimization. Then the direct gradient flow method and unconstrained optimization methods are applied. Numerical experiments show the convergence and accuracy of the proposed methods in both cases, and comparisons on the efficiency are discussed. Finally, the relation between the action and the energy ground states are numerically investigated.
VAFO-Loss: VAscular Feature Optimised Loss Function for Retinal Artery/Vein Segmentation

Estimating clinically-relevant vascular features following vessel segmentation is a standard pipeline for retinal vessel analysis, which provides potential ocular biomarkers for both ophthalmic disease and systemic disease. In this work, we integrate these clinical features into a novel vascular feature optimised loss function (VAFO-Loss), in order to regularise networks to produce segmentation maps, with which more accurate vascular features can be derived. Two common vascular features, vessel density and fractal dimension, are identiﬁed to be sensitive to intra-segment misclassiﬁcation, which is a well-recognised problem in multi-class artery/vein segmentation particularly hindering the estimation of these vascular features. Thus we encode these two features into VAFO-Loss. We ﬁrst show that incorporating our end-to-end VAFO-Loss in standard segmentation networks indeed improves vascular feature estimation, yielding quantitative improvement in stroke incidence prediction, a clinical downstream task. We also report a technically interesting ﬁnd-ing that the trained segmentation network, albeit biased by the feature optimised loss VAFO-Loss, shows statistically signiﬁcant improvement in segmentation metrics, compared to those trained with other state-of-the-art segmentation losses. Code will be released with publication. segmentation · Intra-segment misclassiﬁcation · Vascular feature · Segmentation metrics · Downstream task.
Finite time convergent control using terminal sliding mode

A method for terminal sliding mode control design is discussed. As we know, one of the strong points of terminal sliding mode control is its finite-time convergence to a given equilibrium of the system under consideration, which may be useful in specific applications. The proposed method, different from many existing terminal sliding model control design methods, is studied, and then feedback laws are designed for a class of nonlinear systems, along with illustrative examples.
Bayesian Inference on Hierarchical Nonlocal Priors in Generalized Linear Models

Variable selection methods with nonlocal priors have been widely studied in linear regression models, and their theoretical and empirical performances have been reported. However, the crucial model selection properties for hierarchical nonlocal priors in high-dimensional generalized linear regression have rarely been investigated. In this paper, we consider a hierarchical nonlocal prior for high-dimensional logistic regression models and investigate theoretical properties of the posterior distribution. Specifically, a product moment (pMOM) nonlocal prior is imposed over the regression coefficients with an Inverse-Gamma prior on the tuning parameter. Under standard regularity assumptions, we establish strong model selection consistency in a high-dimensional setting, where the number of covariates is allowed to increase at a subexponential rate with the sample size. We implement the Laplace approximation for computing the posterior probabilities, and a modified shotgun stochastic search procedure is suggested for efficiently exploring the model space. We demonstrate the validity of the proposed method through simulation studies and an RNA-sequencing dataset for stratifying disease risk.
CoNTACT: A Dutch COVID-19 Adapted BERT for Vaccine Hesitancy and Argumentation Detection

We present CoNTACT: a Dutch language model adapted to the domain of COVID-19 tweets. The model was developed by continuing the pre-training phase of RobBERT (Delobelle et al., 2020) by using 2.8M Dutch COVID-19 related tweets posted in 2021. In order to test the performance of the model and compare it to RobBERT, the two models were tested on two tasks: (1) binary vaccine hesitancy detection and (2) detection of arguments for vaccine hesitancy. For both tasks, not only Twitter but also Facebook data was used to show cross-genre performance. In our experiments, CoNTACT showed statistically significant gains over RobBERT in all experiments for task 1. For task 2, we observed substantial improvements in virtually all classes in all experiments. An error analysis indicated that the domain adaptation yielded better representations of domain-specific terminology, causing CoNTACT to make more accurate classification decisions. For task 2, we observed substantial improvements in virtually all classes in all experiments. An error analysis indicated that the domain adaptation yielded better representations of domain-specific terminology, causing CoNTACT to make more accurate classification decisions.