Miracle-dz/UniFAR-FTUs huggingface dataset:
MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks

The field of machine learning (ML) has gained widespread adoption, leading to significant demand for adapting ML to specific scenarios, which is yet expensive and non-trivial. The predominant approaches towards the automation of solving ML tasks (e.g., AutoML) are often time-consuming and hard to understand for human developers. In contrast, though human engineers have the incredible ability to understand tasks and reason about solutions, their experience and knowledge are often sparse and difficult to utilize by quantitative approaches. In this paper, we aim to bridge the gap between machine intelligence and human knowledge by introducing a novel framework MLCopilot, which leverages the state-of-the-art large language models to develop ML solutions for novel tasks. We showcase the possibility of extending the capability of LLMs to comprehend structured inputs and perform thorough reasoning for solving novel ML tasks. And we find that, after some dedicated design, the LLM can (i) observe from the existing experiences of ML tasks and (ii) reason effectively to deliver promising results for new tasks. The solution generated can be used directly to achieve high levels of competitiveness.
ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations

This paper aims to quantitatively evaluate the performance of ChatGPT, an interactive large language model, on inter-sentential relations such as temporal relations, causal relations, and discourse relations. Given ChatGPT's promising performance across various tasks, we proceed to carry out thorough evaluations on the whole test sets of 11 datasets, including temporal and causal relations, PDTB2.0-based, and dialogue-based discourse relations. To ensure the reliability of our findings, we employ three tailored prompt templates for each task, including the zero-shot prompt template, zero-shot prompt engineering (PE) template, and in-context learning (ICL) prompt template, to establish the initial baseline scores for all popular sentence-pair relation classification tasks for the first time. Through our study, we discover that ChatGPT exhibits exceptional proficiency in detecting and reasoning about causal relations, albeit it may not possess the same level of expertise in identifying the temporal order between two events. While it is capable of identifying the majority of discourse relations with existing explicit discourse connectives, the implicit discourse relation remains a formidable challenge. Concurrently, ChatGPT demonstrates subpar performance in the dialogue discourse parsing task that requires structural understanding in a dialogue before being aware of the discourse relation.
Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs

As foundation models continue to exponentially scale in size, efficient methods of adaptation become increasingly critical. Parameter-efficient fine-tuning (PEFT), a recent class of techniques that require only modifying a small percentage of the model parameters, is currently the most popular method for adapting large language models (LLMs). Several PEFT techniques have recently been proposed with varying tradeoffs. We provide a comprehensive and uniform benchmark of various PEFT techniques across a representative LLM, the FLAN-T5 model, and evaluate model performance across different data scales of classification and generation datasets. Based on this, we provide a framework for choosing the optimal fine-tuning techniques given the task type and data availability. Contrary to popular belief, we also empirically prove that PEFT techniques converge slower than full tuning in low data scenarios, and posit the amount of data required for PEFT methods to both perform well and converge efficiently. Lastly, we further optimize these PEFT techniques by selectively choosing which parts of the model to train, and find that these techniques can be applied with significantly fewer parameters while maintaining and even improving performance.
An Analysis of Multiple Particle Swarm Optimizers with Inertia Weight for Multi-objective Optimization

An improved particle swarm optimizer with inertia weight (PSOIW ) was applied to multi-objective optimization (MOO). For further improving its search performance, in this paper, we propose to use a cooperative PSO method called multiple particle swarm optimizers with inertia weight (MPSOIW ) to search. The crucial idea of the MPSOIW , here, is to reinforce the search ability of the PSOIW by the union's power of plural swarms, i.e. distributed processing. To demonstrate the search performance and effect of the proposal, computer experiments on a suite of 2-objective optimization problems are carried out by an aggregation-based manner. The resulting Pareto-optimal solution distributions corresponding to each given problem indicate that the linear weighted aggre- gation among the adopted three kinds of dynamic weighted aggregations is the most suitable for acquiring better search results. Throughout quantitative analysis to experimental data, we clarify the search characteristics and performance effect of the MPSOIW contrast with that of the original PSOIW, PSOIW , and MPSOIW.
Unified Noise-aware Network for Low-count PET Denoising

As PET imaging is accompanied by substantial radiation exposure and cancer risk, reducing radiation dose in PET scans is an important topic. However, low-count PET scans often suffer from high image noise, which can negatively impact image quality and diagnostic performance. Recent advances in deep learning have shown great potential for recovering underlying signal from noisy counterparts. However, neural networks trained on a specific noise level cannot be easily generalized to other noise levels due to different noise amplitude and variances. To obtain optimal denoised results, we may need to train multiple networks using data with different noise levels. But this approach may be infeasible in reality due to limited data availability. Denoising dynamic PET images presents additional challenge due to tracer decay and continuously changing noise levels across dynamic frames. To address these issues, we propose a Unified Noise-aware Network (UNN) that combines multiple sub-networks with varying denoising power to generate optimal denoised results regardless of the input noise levels. Evaluated using large-scale data from two medical centers with different vendors, presented results showed that the UNN can consistently produce promising denoised results regardless of input noise levels, and demonstrate superior performance over networks trained on single noise level data, especially for extremely low-count data.
On The Random Tur\'an number of linear cycles

Given two $r$-uniform hypergraphs $G$ and $H$ the Tur\'an number $\rm{ex}(G, H)$ is the maximum number of edges in an $H$-free subgraph of $G$. We study the typical value of $\rm{ex}(G, H)$ when $G=G_{n,p}^{(r)}$, the Erd\H{o}s-R\'enyi random $r$-uniform hypergraph, and $H=C_{2\ell}^{(r)}$, the $r$-uniform linear cycle of length $2\ell$. The case of graphs ($r=2$) is a longstanding open problem that has been investigated by many researchers. We determine the order of magnitude of $\rm{ex}\left(G_{n,p}^{(r)}, C_{2\ell}^{(r)}\right)$ for all $r\geq 4$ and all $\ell\geq 2$ up to polylogarithmic factors for all values of $p=p(n)$. Our proof is based on the container method and uses a balanced supersaturation result for linear even cycles which improves upon previous such results by Ferber-Mckinley-Samotij and Balogh-Narayanan-Skokan.
Identifiability of causal effects with non-Gaussianity and auxiliary covariates

Assessing causal effects in the presence of unmeasured confounding is a challenging problem. It has been previously shown that the causal effect is identifiable when noise variables of the treatment and outcome are both non-Gaussian under linear model setting with hidden variables. In this paper, we investigate the problem of identifying the causal effect using the auxiliary covariate and non-Gaussianity from the treatment. Our key idea is to characterize the impact of unmeasured confounders using an observed covariate, assuming they are all Gaussian. We demonstrate that the causal effect can be identified using a measured covariate, and then extend the identification results to the multi-treatment setting. We further develop a simple estimation procedure for calculating causal effects.
SAM Meets Robotic Surgery: An Empirical Study in Robustness Perspective

Segment Anything Model (SAM) is a foundation model for semantic segmentation and shows excellent generalization capability with the prompts. In this empirical study, we investigate the robustness and zero-shot generalizability of the SAM in the domain of robotic surgery in various settings of (i) prompted vs. unprompted; (ii) bounding box vs. points-based prompt; (iii) generalization under corruptions and perturbations with five severity levels; and (iv) state-of-the-art supervised model vs. SAM. We conduct all the observations with two well-known robotic instrument segmentation datasets of MICCAI EndoVis 2017 and 2018 challenges. Our extensive evaluation results reveal that although SAM shows remarkable zero-shot generalization ability with bounding box prompts, it struggles to segment the whole instrument with point-based prompts and unprompted settings. Furthermore, our qualitative figures demonstrate that the model either failed to predict the parts of the instrument mask (e.g., jaws, wrist) or predicted parts of the instrument as different classes in the scenario of overlapping instruments within the same bounding box or with the point-based prompt. In fact, it is unable to identify instruments in some complex surgical scenarios of blood, reflection, blur, and shade. Additionally, SAM is insufficiently robust to maintain high performance when subjected to various forms of data corruption. Therefore, we can argue that SAM is not ready for downstream surgical tasks without further domain-specific fine-tuning.
Towards Automated Circuit Discovery for Mechanistic Interpretability

Through considerable effort and intuition, several recent works have reverse-engineered nontrivial behaviors of transformer models. This paper systematizes the mechanistic interpretability process they followed. First, researchers choose a metric and dataset that elicit the desired model behavior. Then, they apply activation patching to find which abstract neural network units are involved in the behavior. By varying the dataset, metric, and units under investigation, researchers can understand the functionality of each component. We automate one of the process' steps: to identify the circuit that implements the specified behavior in the model's computational graph. We propose several algorithms and reproduce previous interpretability results to validate them. For example, the ACDC algorithm rediscovered 5/5 of the component types in a circuit in GPT-2 Small that computes the Greater-Than operation. ACDC selected 68 of the 32,000 edges in GPT-2 Small, all of which were manually found by previous work. Our code is available at https://github.com/ArthurConmy/Automatic-Circuit-Discovery.
Blind Signal Separation for Fast Ultrasound Computed Tomography

Breast cancer is the most prevalent cancer with a high mortality rate in women over the age of 40. Many studies have shown that the detection of cancer at earlier stages significantly reduces patients' mortality and morbidity rages. Ultrasound computer tomography (USCT) is considered as a promising screening tool for diagnosing early-stage breast cancer as it is cost-effective and produces 3D images without radiation exposure. However, USCT is not a popular choice mainly due to its prolonged imaging time. USCT is time-consuming because it needs to transmit a number of ultrasound waves and record them one by one to acquire a high-quality image. We propose FastUSCT, a method to acquire a high-quality image faster than traditional methods for USCT. FastUSCT consists of three steps. First, it transmits multiple ultrasound waves at the same time to reduce the imaging time. Second, it separates the overlapping waves recorded by the receiving elements into each wave with UNet. Finally, it reconstructs an ultrasound image with a synthetic aperture method using the separated waves. We evaluated FastUSCT on simulation on breast digital phantoms. We trained the UNet on simulation using natural images and transferred the model for the breast digital phantoms. The empirical result shows that FastUSCT significantly improves the quality of the image under the same imaging time to the conventional USCT method, especially when the imaging time is limited.
The Hopf Bifurcation of a Predator-prey Model

By using the qualitative theory of the ordinary differential equation,we investigate a predator-prey model of two populations:dx[]dx=b_0x(b_1+b_2x-b_3x~2)-b_4xy,dy[]dt=-cy+(ax-βy)y.The equilibrium points and limit cycles in the model are discussed.It is shown that the model undergoes a Hopf bifurcation when the parameters have the relationship:a_2=(2k-1)+1-2a_1-2k[]x_4,and proved that the periodic solution created by the Hopf bifurcation is stable.
ResiDual: Transformer with Dual Residual Connections

Transformer networks have become the preferred architecture for many tasks due to their state-of-the-art performance. However, the optimal way to implement residual connections in Transformer, which are essential for effective training, is still debated. Two widely used variants are the Post-Layer-Normalization (Post-LN) and Pre-Layer-Normalization (Pre-LN) Transformers, which apply layer normalization after each residual block's output or before each residual block's input, respectively. While both variants enjoy their advantages, they also suffer from severe limitations: Post-LN causes gradient vanishing issue that hinders training deep Transformers, and Pre-LN causes representation collapse issue that limits model capacity. In this paper, we propose ResiDual, a novel Transformer architecture with Pre-Post-LN (PPLN), which fuses the connections in Post-LN and Pre-LN together and inherits their advantages while avoids their limitations. We conduct both theoretical analyses and empirical experiments to verify the effectiveness of ResiDual. Theoretically, we prove that ResiDual has a lower bound on the gradient to avoid the vanishing issue due to the residual connection from Pre-LN. Moreover, ResiDual also has diverse model representations to avoid the collapse issue due to the residual connection from Post-LN. Empirically, ResiDual outperforms both Post-LN and Pre-LN on several machine translation benchmarks across different network depths and data sizes. Thanks to the good theoretical and empirical performance, ResiDual Transformer can serve as a foundation architecture for different AI models (e.g., large language models). Our code is available at https://github.com/microsoft/ResiDual.
The Io, Europa, and Ganymede Auroral Footprints at Jupiter in the Ultraviolet: Positions and Equatorial Lead Angles

Jupiter's satellite auroral footprints are a consequence of the interaction between the Jovian magnetic field with co‐rotating iogenic plasma and the Galilean moons. The disturbances created near the moons propagate as Alfvén waves along the magnetic field lines. The position of the moons is therefore “Alfvénically” connected to their respective auroral footprint. The angular separation from the instantaneous magnetic footprint can be estimated by the so‐called lead angle. That lead angle varies periodically as a function of orbital longitude, since the time for the Alfvén waves to reach the Jovian ionosphere varies accordingly. Using spectral images of the Main Alfvén Wing auroral spots collected by Juno‐UVS during the first 43 orbits, this work provides the first empirical model of the Io, Europa, and Ganymede equatorial lead angles for the northern and southern hemispheres. Alfvén travel times between the three innermost Galilean moons to Jupiter's northern and southern hemispheres are estimated from the lead angle measurements. We also demonstrate the accuracy of the mapping from the Juno magnetic field reference model (JRM33) at the completion of the prime mission for M‐shells extending to at least 15 RJ. Finally, we shows how the added knowledge of the lead angle can improve the interpretation of the moon‐induced decametric emissions.
The mass of TOI-519 b: A close-in giant planet transiting a metal-rich mid-M dwarf

We report on the determination of the mass of TOI-519 b, a transiting substellar object around a mid-M dwarf. We carried out radial velocity measurements using Subaru/InfraRed Doppler (IRD), revealing that TOI-519 b is a planet with a mass of $0.463^{+0.082}_{-0.088}\, M_{\rm Jup}$. We also found that the host star is metal rich ([Fe/H] = 0.27 ± 0.09 dex) and has the lowest effective temperature (Teff = 3322 ± 49 K) among all stars hosting known close-in giant planets based on the IRD spectra and mid-resolution infrared spectra obtained with NASA Infrared Telescope Facility/SpeX. The core mass of TOI-519 b inferred from a thermal evolution model ranges from 0 to ∼30 M⊕, which can be explained by both core accretion and disk instability models as the formation origins of this planet. However, TOI-519 is in line with the emerging trend that M dwarfs with close-in giant planets tend to have high metallicity, which may indicate that they formed in the core accretion model. The system is also consistent with the potential trend that close-in giant planets around M dwarfs tend to be less massive than those around FGK dwarfs.
BiSPARCs for Unsourced Random Access in Massive MIMO

This paper considers the massive MIMO unsourced random access problem in a quasi-static Rayleigh fading setting. The proposed coding scheme is based on a concatenation of a"conventional"channel code (such as, e.g., LDPC) serving as an outer code, and a sparse regression code (SPARC) serving as an inner code. The scheme combines channel estimation, single-user decoding, and successive interference cancellation in a novel way. The receiver performs joint channel estimation and SPARC decoding via an instance of a bilinear generalized approximate message passing (BiGAMP) based algorithm, which leverages the intrinsic bilinear structure that arises in the considered communication regime. The detection step is followed by a per-user soft-input-soft-output (SISO) decoding of the outer channel code in combination with a successive interference cancellation (SIC) step. We show via numerical simulation that the resulting scheme achieves stat-of-the-art performance in the massive connectivity setting, while attaining comparatively low implementation complexity.
Measurements of Correlated Pair Momentum Distributions in 3He(e,e'pp)n with CLAS

We have measured the {sup 3}He(e,e{prime}pp)n reaction at 2.2 and 4.4 GeV over a wide kinematic range. The kinetic energy distribution for ''fast'' nucleons (p > 250 MeV/c) peaks where two nucleons each have 20% or less and the third or ''leading'' nucleon carries most of the transferred energy. These fast nucleon pairs (both pp and pn) are back-to-back and carry very little momentum along {rvec q}, indicating that they are spectators. Experimental and theoretical evidence indicates that we have measured NN correlations in {sup 3}He(e,e{prime}pp)n by striking the third nucleon and detecting the spectator correlated pair.
Earning Extra Performance From Restrictive Feedbacks

Many machine learning applications encounter situations where model providers are required to further refine the previously trained model so as to gratify the specific need of local users. This problem is reduced to the standard model tuning paradigm if the target data is permissibly fed to the model. However, it is rather difficult in a wide range of practical cases where target data is not shared with model providers but commonly some evaluations about the model are accessible. In this paper, we formally set up a challenge named Earning eXtra PerformancE from restriCTive feEDdbacks (EXPECTED) to describe this form of model tuning problems. Concretely, EXPECTED admits a model provider to access the operational performance of the candidate model multiple times via feedback from a local user (or a group of users). The goal of the model provider is to eventually deliver a satisfactory model to the local user(s) by utilizing the feedbacks. Unlike existing model tuning methods where the target data is always ready for calculating model gradients, the model providers in EXPECTED only see some feedbacks which could be as simple as scalars, such as inference accuracy or usage rate. To enable tuning in this restrictive circumstance, we propose to characterize the geometry of the model performance with regard to model parameters through exploring the parameters’ distribution. In particular, for deep models whose parameters distribute across multiple layers, a more query-efficient algorithm is further tailor-designed that conducts layerwise tuning with more attention to those layers which pay off better. Our theoretical analyses justify the proposed algorithms from the aspects of both efficacy and efficiency. Extensive experiments on different applications demonstrate that our work forges a sound solution to the EXPECTED problem, which establishes the foundation for future studies towards this direction.
Rise of Distributed Deep Learning Training in the Big Model Era: From a Software Engineering Perspective

Deep learning (DL) has become a key component of modern software. In the “big model” era, the rich features of DL-based software (i.e., DL software) substantially rely on powerful DL models, e.g., BERT, GPT-3, and the recently emerging GPT-4, which are trained on the powerful cloud with large datasets. Hence, training effective DL models has become a vital stage in the whole software lifecycle. When training deep learning models, especially those big models, developers need to parallelize and distribute the computation and memory resources amongst multiple devices (e.g., a cluster of GPUs) in the training process, which is known as distributed deep learning training, or distributed training for short. However, the unique challenges that developers encounter in distributed training process have not been studied in the software engineering community. Given the increasingly heavy dependence of current DL-based software on distributed training, this paper aims to fill in the knowledge gap and presents the first comprehensive study on developers’ issues in distributed training. To this end, we focus on popular DL frameworks that support distributed training (including TensorFlow, PyTorch, Keras, and Horovod) and analyze 1,131 real-world developers’ issues about using these frameworks reported on Stack Overflow and GitHub. We construct a fine-grained taxonomy consisting of 30 categories regarding the fault symptoms and summarize common fix patterns for different symptoms. We find that: (1) many distributed-specific faults and non-distributed-specific faults inherently share the same fault symptoms, making it challenging to debug; (2) most of the fault symptoms have frequent fix patterns; (3) about half of the faults are related to system-level configurations. Based on the results, we suggest actionable implications on research avenues that can potentially facilitate the distributed training to develop DL-based software, such as focusing on the frequent and common fix patterns when designing testing or debugging tools, developing efficient testing and debugging techniques for communication configuration along with the synthesis of network configuration analysis, designing new multi-device checkpoint-and-replay techniques to help reproduction, and designing serverless APIs for cloud platforms.