Miracle-dz/UniFAR-FTUs huggingface dataset:
Vanishing Curvature in Randomly Initialized Deep ReLU Networks

Deep ReLU networks are at the basis of many modern neural architectures. Yet, the loss landscape of such networks and its interaction with state-of-the-art optimizers is not fully understood. One of the most crucial aspects is the landscape at random initialization, which often influences convergence speed dramatically. In their seminal works, Xavier & Bengio, 2010 and He et al., 2015 propose an initialization strategy that is supposed to prevent gradients from vanishing. Yet, we identify shortcomings of their expectation analysis as network depth increases, and show that the proposed initialization can actually fail to deliver stable gradient norms. More precisely, by leveraging an in-depth analysis of the median of the forward pass, we first show that, with high probability, vanishing gradients cannot be circumvented when the network width scales with less than Ω(depth). Second, we extend this analysis to second-order derivatives and show that random i.i. d. initialization also gives rise to Hessian matrices with eigenspectra that vanish in depth. Whenever this happens, optimizers are initialized in a very flat, saddle pointlike plateau, which is particularly hard to escape with stochastic gradient descent (SGD) as its escaping time is inversely related to curvature magnitudes. We believe that this observation is crucial for fully understanding (a) the historical difficulties of training deep nets with vanilla SGD and (b) the success of adaptive gradient methods, which naturally adapt to curvature and thus quickly escape flat plateaus. Proceedings of the 25 International Conference on Artificial Intelligence and Statistics (AISTATS) 2022, Valencia, Spain. PMLR: Volume 151. Copyright 2022 by the author(s).
A Unified Model of Reasoning and Learning

This paper analyzes the historical development of the conceptions of “reasoning” and “learning”, especially their separation in the study of artiﬁcial intelligence and the attempts to combine them in various ways. A uniﬁed treatment of cognitive functions is provided in the AGI model NARS, where reasoning and learning are diﬀerent facets of the same underlying process.
Adaptively Partitioning Max-Affine Estimators for Convex Regression

This paper considers convex shape-restricted nonparametric regression over subgaussian domain and noise with the squared loss. It introduces a tractable convex piecewise-linear estimator which precomputes a partition of the training data by an adaptive version of farthest-point clustering, approximately ﬁts hyperplanes over the partition cells by minimizing the regularized empirical risk, and projects the result into the max-aﬃne class. The analysis provides an upper bound on the generalization error of this estimator matching the rate of Lipschitz nonparametric regression and proves its adaptivity to the intrinsic dimension of the data mitigating the eﬀect of the curse of dimensionality. The experiments conclude with competitive performance, improved overﬁtting robustness, and signiﬁcant computational savings compared to existing convex regression methods.
On the Value of Prior in Online Learning to Rank

This paper addresses the cold-start problem in online learning to rank (OLTR) . We show both theoretically and empirically that priors improve the quality of ranked lists presented to users interactively based on user feedback. These priors can come in the form of unbiased estimates of the relevance of the ranked items, or more practically, can be obtained from oﬄine-learned models. Our experiments show the eﬀectiveness of priors in improving the short-term regret of tabular OLTR algorithms, based on Thompson sampling and BayesUCB .
Parallel measurement of transcriptomes and proteomes from same single cells using nanodroplet splitting

Single-cell multiomics can provide comprehensive insights into gene regulatory networks, cellular diversity, and temporal dynamics. While tools for co-profiling the single-cell genome, transcriptome, and epigenome are available, accessing the proteome in parallel is more challenging. To overcome this limitation, we developed nanoSPLITS (nanodroplet SPlitting for Linked-multimodal Investigations of Trace Samples), a platform that enables unbiased measurement of the transcriptome and proteome from same single cells using RNA sequencing and mass spectrometry-based proteomics, respectively. We demonstrated the nanoSPLITS can robustly profile > 5000 genes and > 2000 proteins per single cell, and identify cell-type-specific markers from both modalities.
Quantifying visual acuity for pre-clinical testing of visual prostheses

Objective. Visual prostheses currently restore only limited vision. More research and pre-clinical work are required to improve the devices and stimulation strategies that are used to induce neural activity that results in visual perception. Evaluation of candidate strategies and devices requires an objective way to convert measured and modelled patterns of neural activity into a quantitative measure of visual acuity. Approach. This study presents an approach that compares evoked patterns of neural activation with target and reference patterns. A d-prime measure of discriminability determines whether the evoked neural activation pattern is sufficient to discriminate between the target and reference patterns and thus provides a quantified level of visual perception in the clinical Snellen and MAR scales. The magnitude of the resulting value was demonstrated using scaled standardized ‘C’ and ‘E’ optotypes. Main results. The approach was used to assess the visual acuity provided by two alternative stimulation strategies applied to simulated retinal implants with different electrode pitch configurations and differently sized spreads of neural activity. It was found that when there is substantial overlap in neural activity generated by different electrodes, an estimate of acuity based only upon electrode pitch is incorrect; our proposed method gives an accurate result in both circumstances. Significance. Quantification of visual acuity using this approach in pre-clinical development will allow for more rapid and accurate prototyping of improved devices and neural stimulation strategies.
Retina Gap Junction Networks Facilitate Blind Denoising in the Visual Hierarchy

Gap junctions in the retina are electrical synapses, which strength is regulated byambient light conditions. Such tunable synapses are crucial for the denoising function of the early visual system. However, it is unclear that how the plastic gap junction network processes unknown noise, specifically how this process works synergistically with the brain’s higher visual centers. Inspired by the electrically coupled photoreceptors, we develop a computational model of the gap junction filter (G-filter). We show that G-filter is an effective blind denoiser that converts different noise distributions into a similar form. Next, since deep convolutional neural networks (DCNNs) functionally reflect some intrinsic features of the visual cortex, we combine G-filter with DCNNs as retina and ventral visual pathways to investigate the relationship between retinal denoising processing and the brain’s high-level functions. In the image denoising and reconstruction task, G-filter dramatically improve the classic deep denoising convolutional neural network (DnCNN)’s ability to process blind noise. Further, we find that the gap junction strength of the G-filter modulates the receptive field of DnCNN’s output neurons by the Integrated Gradients method. At last, in the image classification task, G-filter strengthens the defense of state-of-the-arts DCNNs (ResNet50, VGG19 and InceptionV3) against blind noise attacks, far exceeding human performance when noise is large. Our results indicate G-filter significantly enhance DCNNs’ ability on various blind denoising tasks, implying an essential role for retina gap junction networks in high-level visual processing.
Finite Sample Analysis of Mean-Volatility Actor-Critic for Risk-Averse Reinforcement Learning

The goal in the standard reinforcement learning problem is to find a policy that optimizes the expected return. However, such an ob-jective is not adequate in a lot of real-life applications, like finance, where controlling the uncertainty of the outcome is imperative. The mean-volatility objective penalizes, through a tunable parameter, policies with high variance of the per-step reward. An interesting property of this objective is that it admits simple linear Bellman equations that resemble, up to a reward transformation, those of the risk-neutral case. However, the required reward transformation is policy-dependent, and requires the (usually unknown) expected return of the used policy. In this work, we propose two general meth-ods for policy evaluation under the mean-volatility objective: the direct method and the factored method. We then extend recent results for finite sample analysis in the risk-neutral actor-critic setting to the mean-volatility case. Our analysis shows that the sample complexity to attain an ϵ -accurate stationary point is the same as that of the risk-neutral version, using either policy evaluation method for training the critic. Finally, we carry out experiments to test the proposed methods in a simple environment that exhibits some trade-off between optimality, in expectation, and uncertainty of outcome.
Identifying novel regulators of placental development using time series transcriptomic data and network analyses

The placenta serves as a connection between the mother and the fetus during pregnancy, and provides the fetus with oxygen, nutrients, and growth hormones. However, the regulatory mechanisms and dynamic gene interaction networks underlying early placental development are understudied. Here, we generated RNA sequencing (RNA-seq) data from mouse fetal placenta tissues at embryonic day (e) 7.5, e8.5 and e9.5 to identify genes with timepoint-specific expression, then inferred gene interaction networks to analyze highly connected network modules. We determined that timepoint-specific gene network modules associated with distinct developmental processes, and with similar expression profiles to specific human placental cell populations. From each module, we obtained hub genes and their direct neighboring genes, which were predicted to govern placental functions. We confirmed that four novel candidate regulators identified through our analyses regulate cell migration in the HTR-8/SVneo cell line. Upon conclusion of this study, we were able to predict several novel regulators of placental development using network analysis of bulk RNA-seq data. Our findings and analysis approaches will be valuable for future studies investigating the transcriptional landscape of early placental development.
Simulation of the effect of the randomness of climbing fiber input on the relationship between motor learning and Focal Task-Specific Dystonia

Focal Task-Specific Dystonia (FTSD) is an intractable neurological disorder with no clear prevention or treatment that affects more than 1% of classical musicians and threatens the active lives of experts due to its task-specific tremor symptoms. In the present study, we focused on the motor learning function of the cerebellum, which has not been the focus of much attention in the past. We numerically simulated the firing of cerebellar Purkinje cells and cerebellar nuclei during eyeblink conditioning as a typical example of cerebellar-related timing motor learning, with the aim to find the principle of the pathogenesis of FTSD at the level of individual neurons. The results showed the sustained firing of cerebellar nuclei after the learning condition in which the climbing fiber input to Purkinje cells was continued randomly. Therefore, the present study claims a suggestive factor regarding the neural mechanism of the cerebellum in the motor learning-induced task-specific tremor, which is a symptom of FTSD. We also proposed a motor learning paradigm, “undesirable motor learning,” in which the motor goal is too advanced to be achieved by repetitions alone and converges to a different result than the desired.
Zidovudine multi-combos with last-line fosfomycin, ceftazidime-avibactam, colistin and tigecycline against Multi-Drug Resistant Klebsiella pneumoniae

Drug repurposing is a novel strategy for the development of new therapies against antibiotic-resistant bacteria. Zidovudine, an antiviral largely used in the HIV-therapy, exerts antibacterial activity against Gram-negative bacteria. Zidovudine was identified in a previous drug repurposing synergy screening as fosfomycin enhancer against Klebsiella pneumoniae ATCC 13883. Our aim was to evaluate the antibacterial in vitro activity of zidovudine-based combinations with last-line antibiotics against MDR/XDR K. pneumoniae isolates. We validated the zidovudine/fosfomycin combination against a collection of 12 MDR K. pneumoniae isolates by the checkerboard assay (CBA). In addition, we performed time-kill assays (TKA) to analyze synergistic and bactericidal activities of zidovudine paired combinations with fosfomycin, ceftazidime-avibactam, colistin and tigecycline. These were compared with frequent clinical combinations in the treatment of MDR Enterobacteriaceae. The potential of the triple zidovudine/fosfomycin/colistin was also assessed by TKA. CBA synergy confirmation rate between zidovudine/fosfomycin was 83.33%. TKA yielded synergy confirmation rates of 83.3% for zidovudine/ceftazidime-avibactam, 75% for zidovudine/fosfomycin, 75% for zidovudine/colistin and 66.6% for zidovudine/tigecycline with potent killing activities. Frequent clinical combinations displayed synergy rates of 41.6% for meropenem/ertapenem, 33.33% for meropenem/colistin, 75% for fosfomycin/colistin and 66.6% for fosfomycin/tigecycline with lower bactericidal efficacy than zidovudine-based combinations. The triple zidovudine/fosfomycin/colistin combination exhibited activities similar to fosfomycin/colistin and fosfomycin/zidovudine. As conclusion, zidovudine is an effective partner in in vitro combinations with existing antibiotics against MDR K. pneumoniae, especially with ceftazidime-avibactam, fosfomycin or colistin. Further studies are needed to elucidate the clinical potential of zidovudine as a repurposed drug in the antibacterial therapy.
On Global-view Based Defense via Adversarial Attack and Defense Risk Guaranteed Bounds

It is well-known that deep neural networks (DNNs) are susceptible to adversarial attacks, which presents the most severe fragility of deep learning system. Despite achieving impressive performance, most of the current state-of-the-art classiﬁers remain highly vulnerable to crafted imperceptible, adversarial perturbations. Recent research attempts to understand neural network attack and defense have become increasingly urgent and important. While rapid progress has been made on this front, there is still an important theoretical gap in achieving guaranteed bounds on attack/defense models, leaving uncertainty in the certiﬁed guarantees of these models. To this end, we systematically address this problem in this paper. More speciﬁcally, we formulate attack and defense in a generic setting where there exists a family of adversaries (i.e., attackers) for attacking a family of classiﬁers (i.e., defenders). We develop a novel class of f-divergences suitable for measuring divergence among multiple distributions. This equips us to study the interactions between attackers and defenders in a countervailing game where we formulate a joint risk on attack and defense schemes. This is followed by our key results on guaranteed upper and lower bounds on this risk that can provide a better understanding of the behaviors of those parties from the attack and defense perspectives, thereby having important implications to both attack and defense sides. Finally, beneﬁted from our theory, we propose an empirical approach that bases
Brain-wide imaging of an adult vertebrate with image transfer oblique plane microscopy

Optical imaging is a powerful tool to visualise and measure neuronal activity. However, due to the size and opacity of vertebrate brains it has until now been impossible to simultaneously image neuronal circuits at cellular resolution across the entire adult brain. This is true even for the smallest known vertebrate brain in the teleost Danionella, which is still too large for existing volumetric imaging approaches. Here we introduce image transfer oblique plane microscopy, which uses a new optical refocusing solution via a custom fibre-optical faceplate, enabling a large field-of-view of up to 4 mm3 at a volume rate of 1 Hz. We demonstrate the power of this method with the first brain-wide recording of neuronal activity in an adult vertebrate.
Substrate adhesion determines migration during mesenchymal cell condensation in chondrogenesis.

Mesenchymal condensation is a prevalent morphogenetic transition, essential in chondrogenesis. However, the current understanding of condensation mechanisms is limited. In vivo, progenitor cells directionally migrate from the surrounding loose mesenchyme towards regions of increasing matrix adherence (the condensation centers), which is accompanied by the upregulation of fibronectin. Here, we focused on the mechanisms of cell migration during mesenchymal cell condensation and the effects of matrix adherence. Dendrimer-based nanopatterns of the cell-adhesive peptide arginine-glycine-aspartic acid (RGD), present in fibronectin were used to regulate substrate adhesion. We recorded single and collective cell migration of mesenchymal stem cells, under chondrogenic induction, by live imaging. Results show that the cell migration mode on single cells depends on substrate adhesiveness, and that cell directionality controls cell condensation and the fusion of condensates. Inhibition experiments showed that N-cadherin-mediated cell-cell interactions are also pivotal for directional migration of cell condensates by maintaining cell-cell cohesion, thus suggesting a fine interplay between cell-matrix and cell-cell adhesions. Results shed light on the role of cell interactions with a fibronectin-depositing matrix during chondrogenesis in vitro, with possible applications in regenerative medicine.
Emergent Low-Frequency Activity in Cortico-Cerebellar Networks with Motor Skill Learning

Abstract The motor cortex controls skilled arm movement by recruiting a variety of targets in the nervous system, and it is important to understand the emergent activity in these regions as refinement of a motor skill occurs. One fundamental projection of the motor cortex (M1) is to the cerebellum. However, the emergent activity in the motor cortex and the cerebellum that appears as a dexterous motor skill is consolidated is incompletely understood. Here, we report on low-frequency oscillatory (LFO) activity that emerges in cortico-cerebellar networks with learning the reach-to-grasp motor skill. We chronically recorded the motor and the cerebellar cortices in rats, which revealed the emergence of coordinated movement-related activity in the local-field potentials as the reaching skill consolidated. Interestingly, we found this emergent activity only in the rats that gained expertise in the task. We found that the local and cross-area spiking activity was coordinated with LFOs in proficient rats. Finally, we also found that these neural dynamics were more prominently expressed during accurate behavior in the M1. This work furthers our understanding on emergent dynamics in the cortico-cerebellar loop that underlie learning and execution of precise skilled movement.
Zygotic genome activation by the totipotency pioneer factor Nr5a2

Life begins with a switch in genetic control from the maternal to the embryonic genome during zygotic genome activation (ZGA) in totipotent embryos. Despite its importance, the essential regulators of ZGA remain largely unknown in mammals. Based on de novo motif searches, we identified the orphan nuclear receptor Nr5a2 as a key activator of major ZGA in mouse embryos. Nr5a2 binds to its motif within a subtype of SINE B1/Alu transposable elements found in cis-regulatory regions of ZGA genes. Chemical inhibition suggests that 72% of ZGA genes are regulated by Nr5a2 and potentially other orphan nuclear family receptors. Consistent with a role in ZGA, Nr5a2 is required for progression beyond the 2-cell stage. Nr5a2 promotes chromatin accessibility during ZGA and binds to entry/exit sites of nucleosomal DNA in vitro. We conclude that Nr5a2 is an essential pioneer factor that distinctly regulates totipotency and pluripotency during mammalian development. One-Sentence Summary Nr5a2 is an essential pioneer transcription factor that activates expression of zygotic genes in mouse embryos.
Construction Vibration Monitoring in the Charleston, South Carolina Area

Vibrations generated during construction often affect adjacent and surrounding buildings and disturb neighboring residents in tightly spaced urban environments. These vibrations can lead to structural damage, especially to older structures. In Charleston, South Carolina, construction vibrations are of special concern due to the tight spacing, age, construction, and historic significance of many of the city’s buildings. Of particular interest are the vibrations generated from pile driving activities. Due to the nature of the lower coastal plain soils in the Charleston, SC area, the majority of new commercial structures are being founded on driven pile foundations bearing within the underlying Cooper Marl Formation. This paper presents the development of vibration threshold levels for both historic and modern structures in Charleston, SC and the case histories of five construction projects in which the developed criteria was used. Vibration data, pre and post-construction surveys, and crack monitoring device data collected during these construction projects were then analyzed to evaluate the vibration criteria.
TCF-7 is dispensable for anti-tumor response but required for persistent function of mature CD8 T cells

T Cell Factor-1, encoded by TCF-7, is a transcription factor that plays an essential role during T cell development and differentiation. In this manuscript we utilized a pre-clinical model provided evidence that TCF-7 is dispensable for the anti-tumor response, and that TCF-7 suppresses key transcriptional factors Eomes and T-bet and molecules responsible for peripheral CD8 T cell cytolytic function. We discovered that TCF-7 regulates NKG2D expression on naïve and activated mouse CD8 T cells, and that peripheral CD8 T cells from TCF-7 cKO utilize NKG2D to clear tumor cells. We also provide evidence that TCF-7 regulates key signaling molecules, including LCK, LAT, ITK, PLC-γ1, P65, ERKI/II, and JAK/STATs required for peripheral CD8 T cell persistent function. Our data transcriptomic and protein data uncovered the mechanism of how TCF-7 impacting peripheral CD8 T cell inflammatory cytokine production, CD8 T cell activation, and apoptosis. Our pre-clinical model showed that CD8 T cells from TCF-7 cKO mice did not cause GVHD, but effectively cleared primary tumor cells.