Miracle-dz/UniFAR-FTUs huggingface dataset:
Knowledge Distillation Performs Partial Variance Reduction

Knowledge distillation is a popular approach for enhancing the performance of ''student'' models, with lower representational capacity, by taking advantage of more powerful ''teacher'' models. Despite its apparent simplicity and widespread use, the underlying mechanics behind knowledge distillation (KD) are still not fully understood. In this work, we shed new light on the inner workings of this method, by examining it from an optimization perspective. We show that, in the context of linear and deep linear models, KD can be interpreted as a novel type of stochastic variance reduction mechanism. We provide a detailed convergence analysis of the resulting dynamics, which hold under standard assumptions for both strongly-convex and non-convex losses, showing that KD acts as a form of partial variance reduction, which can reduce the stochastic gradient noise, but may not eliminate it completely, depending on the properties of the ''teacher'' model. Our analysis puts further emphasis on the need for careful parametrization of KD, in particular w.r. t. the weighting of the distillation loss, and is validated empirically on both linear models and deep neural networks.
Error Estimation for Single-Image Human Body Mesh Reconstruction

Human pose and shape estimation methods continue to suffer in situations where one or more parts of the body are occluded. More importantly, these methods cannot express when their predicted pose is incorrect. This has serious consequences when these methods are used in human-robot interaction scenarios, where we need methods that can evaluate their predictions and flag situations where they might be wrong. This work studies this problem. We propose a method that combines information from OpenPose and SPIN -- two popular human pose and shape estimation methods -- to highlight regions on the predicted mesh that are least reliable. We have evaluated the proposed approach on 3DPW, 3DOH, and Human3.6M datasets, and the results demonstrate our model's effectiveness in identifying inaccurate regions of the human body mesh. Our code is available at https://github.com/Hamoon1987/meshConfidence.
Pruning at Initialization - A Sketching Perspective

The lottery ticket hypothesis (LTH) has increased attention to pruning neural networks at initialization. We study this problem in the linear setting. We show that finding a sparse mask at initialization is equivalent to the sketching problem introduced for efficient matrix multiplication. This gives us tools to analyze the LTH problem and gain insights into it. Specifically, using the mask found at initialization, we bound the approximation error of the pruned linear model at the end of training. We theoretically justify previous empirical evidence that the search for sparse networks may be data independent. By using the sketching perspective, we suggest a generic improvement to existing algorithms for pruning at initialization, which we show to be beneficial in the data-independent case.
Local geometry of NAE-SAT solutions in the condensation regime

The local behavior of typical solutions of random constraint satisfaction problems (CSP) describes many important phenomena including clustering thresholds, decay of correlations, and the behavior of message passing algorithms. When the constraint density is low, studying the planted model is a powerful technique for determining this local behavior which in many examples has a simple Markovian structure. Work of Coja-Oghlan, Kapetanopoulos, M\"{u}ller (2020) showed that for a wide class of models, this description applies up to the so-called condensation threshold. Understanding the local behavior after the condensation threshold is more complex due to long-range correlations. In this work, we revisit the random regular NAE-SAT model in the condensation regime and determine the local weak limit which describes a random solution around a typical variable. This limit exhibits a complicated non-Markovian structure arising from the space of solutions being dominated by a small number of large clusters. This is the first description of the local weak limit in the condensation regime for any sparse random CSPs in the one-step replica symmetry breaking (1RSB) class. Our result is non-asymptotic, and characterizes the tight fluctuation $O(n^{-1/2})$ around the limit. Our proof is based on coupling the local neighborhoods of an infinite spin system, which encodes the structure of the clusters, to a broadcast model on trees whose channel is given by the 1RSB belief-propagation fixed point. We believe that our proof technique has broad applicability to random CSPs in the 1RSB class.
Multi-Modal Wireless Flexible Gel-Free Sensors with Edge Deep Learning for Detecting and Alerting Freezing of Gait in Parkinson's Patients

Freezing of gait (FoG) is a debilitating symptom of Parkinson's disease (PD). This work develops flexible wearable sensors that can detect FoG and alert patients and companions to help prevent falls. FoG is detected on the sensors using a deep learning (DL) model with multi-modal sensory inputs collected from distributed wireless sensors. Two types of wireless sensors are developed, including: (1) a C-shape central node placed around the patient's ears, which collects electroencephalogram (EEG), detects FoG using an on-device DL model, and generates auditory alerts when FoG is detected; (2) a stretchable patch-type sensor attached to the patient's legs, which collects electromyography (EMG) and movement information from accelerometers. The patch-type sensors wirelessly send collected data to the central node through low-power ultra-wideband (UWB) transceivers. All sensors are fabricated on flexible printed circuit boards. Adhesive gel-free acetylene carbon black and polydimethylsiloxane electrodes are fabricated on the flexible substrate to allow conformal wear over the long term. Custom integrated circuits (IC) are developed in 180 nm CMOS technology and used in both types of sensors for signal acquisition, digitization, and wireless communication. A novel lightweight DL model is trained using multi-modal sensory data. The inference of the DL model is performed on a low-power microcontroller in the central node. The DL model achieves a high detection sensitivity of 0.81 and a specificity of 0.88. The developed wearable sensors are ready for clinical experiments and hold great promise in improving the quality of life of patients with PD. The proposed design methodologies can be used in wearable medical devices for the monitoring and treatment of a wide range of neurodegenerative diseases.
NASimEmu: Network Attack Simulator & Emulator for Training Agents Generalizing to Novel Scenarios

Current frameworks for training offensive penetration testing agents with deep reinforcement learning struggle to produce agents that perform well in real-world scenarios, due to the reality gap in simulation-based frameworks and the lack of scalability in emulation-based frameworks. Additionally, existing frameworks often use an unrealistic metric that measures the agents' performance on the training data. NASimEmu, a new framework introduced in this paper, addresses these issues by providing both a simulator and an emulator with a shared interface. This approach allows agents to be trained in simulation and deployed in the emulator, thus verifying the realism of the used abstraction. Our framework promotes the development of general agents that can transfer to novel scenarios unseen during their training. For the simulation part, we adopt an existing simulator NASim and enhance its realism. The emulator is implemented with industry-level tools, such as Vagrant, VirtualBox, and Metasploit. Experiments demonstrate that a simulation-trained agent can be deployed in emulation, and we show how to use the framework to train a general agent that transfers into novel, structurally different scenarios. NASimEmu is available as open-source.
Faster Margin Maximization Rates for Generic Optimization Methods

First-order optimization methods tend to inherently favor certain solutions over others when minimizing a given training objective with multiple local optima. This phenomenon, known as implicit bias, plays a critical role in understanding the generalization capabilities of optimization algorithms. Recent research has revealed that gradient-descent-based methods exhibit an implicit bias for the $\ell_2$-maximal margin classifier in the context of separable binary classification. In contrast, generic optimization methods, such as mirror descent and steepest descent, have been shown to converge to maximal margin classifiers defined by alternative geometries. However, while gradient-descent-based algorithms demonstrate fast implicit bias rates, the implicit bias rates of generic optimization methods have been relatively slow. To address this limitation, in this paper, we present a series of state-of-the-art implicit bias rates for mirror descent and steepest descent algorithms. Our primary technique involves transforming a generic optimization algorithm into an online learning dynamic that solves a regularized bilinear game, providing a unified framework for analyzing the implicit bias of various optimization methods. The accelerated rates are derived leveraging the regret bounds of online learning algorithms within this game framework.
Optimization’s Neglected Normative Commitments

Optimization is offered as an objective approach to resolving complex, real-world decisions involving uncertainty and conflicting interests. It drives business strategies as well as public policies and, increasingly, lies at the heart of sophisticated machine learning systems. A paradigm used to approach potentially high-stakes decisions, optimization relies on abstracting the real world to a set of decision(s), objective(s) and constraint(s). Drawing from the modeling process and a range of actual cases, this paper describes the normative choices and assumptions that are necessarily part of using optimization. It then identifies six emergent problems that may be neglected: 1) Misspecified values can yield optimizations that omit certain imperatives altogether or incorporate them incorrectly as a constraint or as part of the objective, 2) Problematic decision boundaries can lead to faulty modularity assumptions and feedback loops, 3) Failing to account for multiple agents’ divergent goals and decisions can lead to policies that serve only certain narrow interests, 4) Mislabeling and mismeasurement can introduce bias and imprecision, 5) Faulty use of relaxation and approximation methods, unaccompanied by formal characterizations and guarantees, can severely impede applicability, and 6) Treating optimization as a justification for action, without specifying the necessary contextual information, can lead to ethically dubious or faulty decisions. Suggestions are given to further understand and curb the harms that can arise when optimization is used wrongfully.
Synthesizing a Progression of Subtasks for Block-Based Visual Programming Tasks

Block-based visual programming environments play an increasingly important role in introducing computing concepts to K-12 students. In recent years, they have also gained popularity in neuro-symbolic AI, serving as a benchmark to evaluate general problem-solving and logical reasoning skills. The open-ended and conceptual nature of these visual programming tasks make them challenging, both for state-of-the-art AI agents as well as for novice programmers. A natural approach to providing assistance for problem-solving is breaking down a complex task into a progression of simpler subtasks; however, this is not trivial given that the solution codes are typically nested and have non-linear execution behavior. In this paper, we formalize the problem of synthesizing such a progression for a given reference block-based visual programming task. We propose a novel synthesis algorithm that generates a progression of subtasks that are high-quality, well-spaced in terms of their complexity, and solving this progression leads to solving the reference task. We show the utility of our synthesis algorithm in improving the efficacy of AI agents (in this case, neural program synthesizers) for solving tasks in the Karel programming environment. Then, we conduct a user study to demonstrate that our synthesized progression of subtasks can assist a novice programmer in solving tasks in the Hour of Code: Maze Challenge by Code-dot-org.
Python Wrapper for Simulating Multi-Fidelity Optimization on HPO Benchmarks without Any Wait

Hyperparameter (HP) optimization of deep learning (DL) is essential for high performance. As DL often requires several hours to days for its training, HP optimization (HPO) of DL is often prohibitively expensive. This boosted the emergence of tabular or surrogate benchmarks, which enable querying the (predictive) performance of DL with a specific HP configuration in a fraction. However, since the actual runtime of a DL training is significantly different from its query response time, simulators of an asynchronous HPO, e.g. multi-fidelity optimization, must wait for the actual runtime at each iteration in a na\"ive implementation; otherwise, the evaluation order during simulation does not match with the real experiment. To ease this issue, we developed a Python wrapper and describe its usage. This wrapper forces each worker to wait so that we yield exactly the same evaluation order as in the real experiment with only $10^{-2}$ seconds of waiting instead of waiting several hours. Our implementation is available at https://github.com/nabenabe0928/mfhpo-simulator/.
Anderson Localization for Schr\"odinger Operators with Monotone Potentials over Circle Homeomorphisms

In this paper, we prove pure point spectrum for a large class of Schr\"odinger operators over circle maps with conditions on the rotation number going beyond the Diophantine. More specifically, we develop the scheme to obtain pure point spectrum for Schr\"odinger operators with monotone bi-Lipschitz potentials over orientation-preserving circle homeomorphisms with Diophantine or weakly Liouville rotation number. The localization is uniform when the coupling constant is large enough.
Efficient Quantum Work Reservoirs at the Nanoscale

When reformulated as a resource theory, thermodynamics can analyze system behaviors in the single-shot regime. In this, the work required to implement state transitions is bounded by $\alpha$-Renyi divergences and so differs in identifying efficient operations compared to stochastic thermodynamics. Thus, a detailed understanding of the difference between stochastic thermodynamics and resource-theoretic thermodynamics is needed. To this end, we study reversibility in the single-shot regime, generalizing the two-level work reservoirs used there to multi-level work reservoirs. This achieves reversibility in any transition in the single-shot regime. Building on this, we systematically explore multi-level work reservoirs in the nondissipation regime with and without catalysts. The resource-theoretic results show that two-level work reservoirs undershoot Landauer's bound, misleadingly implying energy dissipation during computation. In contrast, we demonstrate that multi-level work reservoirs achieve Landauer's bound and produce zero entropy.
Effective health and wellness systems for rural and remote Indigenous communities: a rapid review

Abstract Background: The Canadian healthcare system bares a long legacy of colonisation and assimilation of Indigenous values and approaches to health and wellness. This system often perpetuates social and health inequities through systemic racism, underfunding, lack of culturally appropriate care and barriers to access care. Current funding legislation policies enacted across federal, provincialand territorial governments do not necessarily uphold Indigenous Peoples’ rights to self-determination, health and wellness. We summarise literature on promising Indigenous health systems and practices that prioritise and/or improve rural Indigenous Peoples’ health and wellness. Objective: The impetus for this review was to provide information on promising health systems, while Dehcho First Nations developed a health and wellness vision. Methods: Documents were gathered from indexed and non-indexed databases to obtain literature from peer-reviewed and non-peer reviewed sources. Two reviewers independently 1) screened titles, abstracts and full texts to ensure they met the inclusion criteria, 2) gathered relevant data from all included documents and 3) identified major themes and sub-themes. Reviewers then discussed and reached consensus on the themes. Results: Thematic analysis revealed six themes for effective health systems for rural and remote Indigenous communities: 1) access to primary care, 2) multi-directional knowledge exchange, 3) culturally appropriate care, 4) training and building community capacity, 5) integrated care and 6) health system funding. Conclusion: Effective health and wellness systems must support Indigenous ways of knowing and doing in healthcare models based on collaborative partnerships with community members, health providers and government agencies. summary Sustainable health and wellness funding plans must be provided through federal, provincial, territorial and state partnerships. Indigenous core values, culture and knowledge must be integrated within mainstream health systems. Indigenous Rightsholders must inform all plans for altering or implementing Indigenous health systems. Non-Indigenous cultural safety and competency training can improve care.
Investigation of the relationship between serum irisin level in the idiopathic restless legs syndrome: Could be a marker independent of physical activity?

Restless legs syndrome (RLS) is a common but underdiagnosed neurological syndrome. It is characterized by the feeling of discomfort and desire to move, especially in the lower extremities, which often occurs at night, and the cure or relief of symptoms with movement. Irisin is a hormonelike polypeptide that was first identified in 2012, weighs 22 kDa, consists of 163 amino acids, and is mainly synthesized in muscle. Its synthesis increases with exercise. Here in this study, we planned to investigate the relationship among serum irisin level, physical activity, lipid profile, and RLS.
The neural signature of reality‐monitoring: A meta‐analysis of functional neuroimaging studies

Distinguishing imagination and thoughts from information we perceived from the environment, a process called reality‐monitoring, is important in everyday situations. Although reality monitoring seems to overlap with the concept of self‐monitoring, which allows one to distinguish self‐generated actions or thoughts from those generated by others, the two concepts remain largely separate cognitive domains and their common brain substrates have received little attention. We investigated the brain regions involved in these two cognitive processes and explored the common brain regions they share. To do this, we conducted two separate coordinate‐based meta‐analyses of functional magnetic resonance imaging studies assessing the brain regions involved in reality‐ and self‐monitoring. Few brain regions survived threshold‐free cluster enhancement family‐wise multiple comparison correction (p < .05), likely owing to the small number of studies identified. Using uncorrected statistical thresholds recommended by Signed Differential Mapping with Permutation of Subject Images, the meta‐analysis of reality‐monitoring studies (k = 9 studies including 172 healthy subjects) revealed clusters in the lobule VI of the cerebellum, the right anterior medial prefrontal cortex and anterior thalamic projections. The meta‐analysis of self‐monitoring studies (k = 12 studies including 192 healthy subjects) highlighted the involvement of a set of brain regions including the lobule VI of the left cerebellum and fronto‐temporo‐parietal regions. We showed with a conjunction analysis that the lobule VI of the cerebellum was consistently engaged in both reality‐ and self‐monitoring. The current findings offer new insights into the common brain regions underlying reality‐monitoring and self‐monitoring, and suggest that the neural signature of the self that may occur during self‐production should persist in memories.
Putative adaptive loci show parallel clinal variation in a California‐endemic wildflower

As the global climate crisis continues, predictions concerning how wild populations will respond to changing climate conditions are informed by an understanding of how populations have responded and/or adapted to climate variables in the past. Changes in the local biotic and abiotic environment can drive differences in phenology, physiology, morphology and demography between populations leading to local adaptation, yet the molecular basis of adaptive evolution in wild non‐model organisms is poorly understood. We leverage comparisons between two lineages of Calochortus venustus occurring along parallel transects that allow us to identify loci under selection and measure clinal variation in allele frequencies as evidence of population‐specific responses to selection along climatic gradients. We identify targets of selection by distinguishing loci that are outliers to population structure and by using genotype–environment associations across transects to detect loci under selection from each of nine climatic variables. Despite gene flow between individuals of different floral phenotypes and between populations, we find evidence of ecological specialization at the molecular level, including genes associated with key plant functions linked to plant adaptation to California's Mediterranean climate. Single‐nucleotide polymorphisms (SNPs) present in both transects show similar trends in allelic similarity across latitudes indicating parallel adaptation to northern climates. Comparisons between eastern and western populations across latitudes indicate divergent genetic evolution between transects, suggesting local adaptation to either coastal or inland habitats. Our study is among the first to show repeated allelic variation across climatic clines in a non‐model organism.
The Use of Photoplethysmography in the Assessment of Mental Health: Scoping Review

Background With the rise in mental health problems globally, mobile health provides opportunities for timely medical care and accessibility. One emerging area of mobile health involves the use of photoplethysmography (PPG) to assess and monitor mental health. Objective In recent years, there has been an increase in the use of PPG-based technology for mental health. Therefore, we conducted a review to understand how PPG has been evaluated to assess a range of mental health and psychological problems, including stress, depression, and anxiety. Methods A scoping review was performed using PubMed and Google Scholar databases. Results A total of 24 papers met the inclusion criteria and were included in this review. We identified studies that assessed mental health via PPG using finger- and face-based methods as well as smartphone-based methods. There was variation in study quality. PPG holds promise as a potential complementary technology for detecting changes in mental health, including depression and anxiety. However, rigorous validation is needed in diverse clinical populations to advance PPG technology in tackling mental health problems. Conclusions PPG holds promise for assessing mental health problems; however, more research is required before it can be widely recommended for clinical use.