Miracle-dz/UniFAR-FTUs huggingface dataset:
Module-wise Adaptive Distillation for Multimodality Foundation Models

Pre-trained multimodal foundation models have demonstrated remarkable generalizability but pose challenges for deployment due to their large sizes. One effective approach to reducing their sizes is layerwise distillation, wherein small student models are trained to match the hidden representations of large teacher models at each layer. Motivated by our observation that certain architecture components, referred to as modules, contribute more significantly to the student's performance than others, we propose to track the contributions of individual modules by recording the loss decrement after distillation each module and choose the module with a greater contribution to distill more frequently. Such an approach can be naturally formulated as a multi-armed bandit (MAB) problem, where modules and loss decrements are considered as arms and rewards, respectively. We then develop a modified-Thompson sampling algorithm named OPTIMA to address the nonstationarity of module contributions resulting from model updating. Specifically, we leverage the observed contributions in recent history to estimate the changing contribution of each module and select modules based on these estimations to maximize the cumulative contribution. We evaluate the effectiveness of OPTIMA through distillation experiments on various multimodal understanding and image captioning tasks, using the CoCa-Large model (Yu et al., 2022) as the teacher model.
WAIT: Feature Warping for Animation to Illustration video Translation using GANs

In this paper, we explore a new domain for video-to-video translation. Motivated by the availability of animation movies that are adopted from illustrated books for children, we aim to stylize these videos with the style of the original illustrations. Current state-of-the-art video-to-video translation models rely on having a video sequence or a single style image to stylize an input video. We introduce a new problem for video stylizing where an unordered set of images are used. This is a challenging task for two reasons: i) we do not have the advantage of temporal consistency as in video sequences; ii) it is more difficult to obtain consistent styles for video frames from a set of unordered images compared to using a single image. Most of the video-to-video translation methods are built on an image-to-image translation model, and integrate additional networks such as optical flow, or temporal predictors to capture temporal relations. These additional networks make the model training and inference complicated and slow down the process. To ensure temporal coherency in video-to-video style transfer, we propose a new generator network with feature warping layers which overcomes the limitations of the previous methods. We show the effectiveness of our method on three datasets both qualitatively and quantitatively. Code and pretrained models are available at https://github.com/giddyyupp/wait.
Sub-quadratic (1+\eps)-approximate Euclidean Spanners, with Applications

We study graph spanners for point-set in the high-dimensional Euclidean space. On the one hand, we prove that spanners with stretch<\sqrt{2} and subquadratic size are not possible, even if we add Steiner points. On the other hand, if we add extra nodes to the graph (non-metric Steiner points), then we can obtain (1+\eps)-approximate spanners of subquadratic size. We show how to construct a spanner of size n^{2-\Omega(\eps^3)}, as well as a directed version of the spanner of size n^{2-\Omega(\eps^2)}. We use our directed spanner to obtain an algorithm for computing (1+\eps)-approximation to Earth-Mover Distance (optimal transport) between two sets of size n in time n^{2-\Omega(\eps^2)}.
Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models

Despite the remarkable success of Large Language Models (LLMs), the massive size poses significant deployment challenges, particularly on resource-constrained hardware. While existing LLM compression methods focus on quantization, pruning remains relatively unexplored due to the high cost of training-based approaches and data collection challenges. One-shot pruning methods, although cost-effective and data-free, have become dominant in LLM pruning, but lead to performance decline under the structured pruning setting. In this work, we introduce a new paradigm for structurally pruning LLMs, called Compresso. Our approach, through the collaboration of the proposed resource-efficient pruning algorithm and the LLM itself, learns optimal pruning decisions during the training process. Compresso addresses the challenges of expensive training costs and data collection by incorporating Low-Rank Adaptation (LoRA) into the $L_0$ regularization during the instruction tuning process. Then, we further augment the pruning algorithm by introducing a collaborative prompt that fosters collaboration between the LLM and the pruning algorithm, significantly boosting the overall performance. To this end, Compresso prunes LLaMA-7B to 5.4B, maintaining original performance and even surpassing LLaMA-7B in reading comprehension by 2.62%. Extensive experiments demonstrate that Compresso significantly outperforms one-shot pruning baselines across various sparsity ratios, achieving up to 2.21%, 11.43%, 7.04%, and 4.81% higher scores on the commonsense reasoning, reading comprehension, MMLU, and BBH benchmarks, respectively.
Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT and GPT-4 for Mining Insights at Scale

Large language models (LLMs), such as ChatGPT and GPT-4, are gaining wide-spread real world use. Yet, these LLMs are closed source, and little is known about their performance in real-world use cases. In this paper, we apply and evaluate the combination of ChatGPT and GPT-4 for the real-world task of mining insights from a text corpus in order to identify research challenges in the field of HCI. We extract 4,392 research challenges in over 100 topics from the 2023 CHI conference proceedings and visualize the research challenges for interactive exploration. We critically evaluate the LLMs on this practical task and conclude that the combination of ChatGPT and GPT-4 makes an excellent cost-efficient means for analyzing a text corpus at scale. Cost-efficiency is key for flexibly prototyping research ideas and analyzing text corpora from different perspectives, with implications for applying LLMs for mining insights in academia and practice.
TopicAdapt- An Inter-Corpora Topics Adaptation Approach

Topic models are popular statistical tools for detecting latent semantic topics in a text corpus. They have been utilized in various applications across different fields. However, traditional topic models have some limitations, including insensitivity to user guidance, sensitivity to the amount and quality of data, and the inability to adapt learned topics from one corpus to another. To address these challenges, this paper proposes a neural topic model, TopicAdapt, that can adapt relevant topics from a related source corpus and also discover new topics in a target corpus that are absent in the source corpus. The proposed model offers a promising approach to improve topic modeling performance in practical scenarios. Experiments over multiple datasets from diverse domains show the superiority of the proposed model against the state-of-the-art topic models.
Intelligent Tutoring System: Experience of Linking Software Engineering and Programming Teaching

The increasing number of computer science students pushes lecturers and tutors of first-year programming courses to their limits to provide high-quality feedback to the students. Existing systems that handle automated grading primarily focus on the automation of test case executions in the context of programming assignments. However, they cannot provide customized feedback about the students' errors, and hence, cannot replace the help of tutors. While recent research works in the area of automated grading and feedback generation address this issue by using automated repair techniques, so far, to the best of our knowledge, there has been no real-world deployment of such techniques. Based on the research advances in recent years, we have built an intelligent tutoring system that has the capability of providing automated feedback and grading. Furthermore, we designed a Software Engineering course that guides third-year undergraduate students in incrementally developing such a system over the coming years. Each year, students will make contributions that improve the current implementation, while at the same time, we can deploy the current system for usage by first year students. This paper describes our teaching concept, the intelligent tutoring system architecture, and our experience with the stakeholders. This software engineering project for the students has the key advantage that the users of the system are available in-house (i.e., students, tutors, and lecturers from the first-year programming courses). This helps organize requirements engineering sessions and builds awareness about their contribution to a"to be deployed"software project. In this multi-year teaching effort, we have incrementally built a tutoring system that can be used in first-year programming courses. Further, it represents a platform that can integrate the latest research results in APR for education.
Categorical Landau Paradigm for Gapped Phases

We propose a unified framework to classify gapped infra-red (IR) phases with categorical symmetries, leading to a generalized, categorical Landau paradigm. This is applicable in any dimension and gives a succinct, comprehensive, and computationally powerful approach to classifying gapped symmetric phases. The key tool is the symmetry topological field theory (SymTFT), which is a one dimension higher TFT with two boundaries, which we choose both to be topological. We illustrate the general idea for (1+1)d gapped phases with categorical symmetries and suggest higher-dimensional extensions.
In Vitro and In Silico Characterization of the Aggregation of Thrombi on Ventricular Assist Device Cannula

The unacceptably high stroke rate of HeartMate III VAD without signs of adherent pump thrombosis is hypothesized to be the result of the thrombi originating on the inflow cannula, ingesting and ejecting emboli from the VAD. Therefore, inflow cannula thrombosis has been an emerging focus. The inflow cannula of contemporary VADs, which incorporate both polished and rough regions serve as useful benchmarks to study the effects of roughness and shear on thrombogenesis. An in vitro study was conducted to emulate the micro-hemodynamic condition on a sintered inflow cannula, and to observe the deposition and detachment patterns. Together with a computational fluid dynamic tool, this study aimed to provide insight into the optimization of inflow cannula and potentially reducing adverse neurological events due to upstream thrombus.
Hierarchical Unsupervised Topological SLAM

In this paper we present a novel framework for unsupervised topological clustering resulting in improved loop detection and closure for SLAM. A navigating mobile robot clusters its traversal into visually similar topologies where each cluster (topology) contains a set of similar looking images typically observed from spatially adjacent locations. Each such set of spatially adjacent and visually similar grouping of images constitutes a topology obtained without any supervision. We formulate a hierarchical loop discovery strategy that first detects loops at the level of topologies and subsequently at the level of images between the looped topologies. We show over a number of traversals across different Habitat environments that such a hierarchical pipeline significantly improves SOTA image based loop detection and closure methods. Further, as a consequence of improved loop detection, we enhance the loop closure and backend SLAM performance. Such a rendering of a traversal into topological segments is beneficial for downstream tasks such as navigation that can now build a topological graph where spatially adjacent topological clusters are connected by an edge and navigate over such topological graphs.
Deep learning modelling of manufacturing and build variations on multi-stage axial compressors aerodynamics

Application of deep learning methods to physical simulations such as CFD (Computational Fluid Dynamics) for turbomachinery applications, have been so far of limited industrial relevance. This paper demonstrates the development and application of a deep learning framework for real-time predictions of the impact of manufacturing and build variations, such as tip clearance and surface roughness, on the flow field and aerodynamic performance of multi-stage axial compressors in gas turbines. The associated scatter in compressor efficiency is known to have a significant impact on the corresponding overall performance and emissions of the gas turbine, therefore posing a challenge of great industrial and environmental relevance. The proposed architecture is proven to achieve an accuracy comparable to that of the CFD benchmark, in real-time, for an industrially relevant application. The deployed model, is readily integrated within the manufacturing and build process of gas turbines, thus providing the opportunity to analytically assess the impact on performance and potentially reduce requirements for expensive physical tests.
Evidence of Ultrafaint Radio Frequency Interference in Deep 21 cm Epoch of Reionization Power Spectra with the Murchison Wide-field Array

We present deep upper limits from the 2014 Murchison Widefield Array Phase I observing season, with a particular emphasis on identifying the spectral fingerprints of extremely faint radio frequency interference (RFI) contamination in the 21 cm power spectra (PS). After meticulous RFI excision involving a combination of the SSINS RFI flagger and a series of PS-based jackknife tests, our lowest upper limit on the Epoch of Reionization (EoR) 21 cm PS signal is Δ2 ≤ 1.61 × 104 mK2 at k = 0.258h Mpc−1 at a redshift of 7.1 using 14.7 hr of data. By leveraging our understanding of how even fainter RFI is likely to contaminate the EoR PS, we are able to identify ultrafaint RFI signals in the cylindrical PS. Surprisingly this signature is most obvious in PS formed with less than 1 hr of data, but is potentially subdominant to other systematics in multiple-hour integrations. Since the total RFI budget in a PS detection is quite strict, this nontrivial integration behavior suggests a need to more realistically model coherently integrated ultrafaint RFI in PS measurements so that its potential contribution to a future detection can be diagnosed.
HalluciDet: Hallucinating RGB Modality for Person Detection Through Privileged Information

A powerful way to adapt a visual recognition model to a new domain is through image translation. However, common image translation approaches only focus on generating data from the same distribution as the target domain. Given a cross-modal application, such as pedestrian detection from aerial images, with a considerable shift in data distribution between infrared (IR) to visible (RGB) images, a translation focused on generation might lead to poor performance as the loss focuses on irrelevant details for the task. In this paper, we propose HalluciDet, an IR-RGB image translation model for object detection. Instead of focusing on reconstructing the original image on the IR modality, it seeks to reduce the detection loss of an RGB detector, and therefore avoids the need to access RGB data. This model produces a new image representation that enhances objects of interest in the scene and greatly improves detection performance. We empirically compare our approach against state-of-the-art methods for image translation and for fine-tuning on IR, and show that our HalluciDet improves detection accuracy in most cases by exploiting the privileged information encoded in a pre-trained RGB detector. Code: https://github.com/heitorrapela/HalluciDet
Reinforcement Learning with Fast and Forgetful Memory

Nearly all real world tasks are inherently partially observable, necessitating the use of memory in Reinforcement Learning (RL). Most model-free approaches summarize the trajectory into a latent Markov state using memory models borrowed from Supervised Learning (SL), even though RL tends to exhibit different training and efficiency characteristics. Addressing this discrepancy, we introduce Fast and Forgetful Memory, an algorithm-agnostic memory model designed specifically for RL. Our approach constrains the model search space via strong structural priors inspired by computational psychology. It is a drop-in replacement for recurrent neural networks (RNNs) in recurrent RL algorithms, achieving greater reward than RNNs across various recurrent benchmarks and algorithms without changing any hyperparameters. Moreover, Fast and Forgetful Memory exhibits training speeds two orders of magnitude faster than RNNs, attributed to its logarithmic time and linear space complexity. Our implementation is available at https://github.com/proroklab/ffm.
Representative Days and Hours with Piecewise Linear Transitions for Power System Planning

Electric demand and renewable power are highly variable, and the solution of a planning model relies on capturing this variability. This paper proposes a hybrid multi-area method that effectively captures both the intraday and interday chronology of real data considering extreme values, using a limited number of representative days, and time points within each day. An optimization-based representative extraction method is proposed to improve intraday chronology capturing. It ensures higher precision in preserving data chronology and extreme values than hierarchical clustering methods. The proposed method is based on a piecewise linear demand and supply representation, which reduces approximation errors compared to the traditional piecewise constant formulation. Additionally, sequentially linked day blocks with identical representatives, created through a mapping process, are employed for interday chronology capturing. To evaluate the efficiency of the proposed method, a comprehensive expansion co-planning model is developed, including transmission lines, energy storage systems, and wind farms.
Numerical simulations of the latest caldera-forming eruption of Okmok volcano, Alaska

The latest caldera-forming eruption of Okmok volcano, Alaska, had a global atmospheric impact with tephra deposits found in distant Arctic ice cores and a sulfate signal found in Antarctic ice cores. The associated large-scale climate cooling was driven by the amount of sulfur injected into the stratosphere during the climactic phase of the eruption. This phase was dominated by pyroclastic density currents, which have complex emplacement dynamics precluding direct estimates of the sulfur stratospheric load. We simulated the dynamics of this climactic phase with the two-phase flow model MFIX-TFM under axisymmetric conditions with several combinations of injection mass flux, emission duration, and topography. Results suggest that a steady mass flux of $8.6-28\times 10^9$ kg/s is consistent with field observations. Stratospheric injections occur in pulses issued from 1) the central plume initially rising above the caldera center, 2) successive co-ignimbrite clouds caused by the encounter of the pyroclastic density currents with topography, and 3) the buoyant lift-off of dilute parts of the currents at the end of the eruption. Overall, 2.5 to 25% of the emitted volcanic gas reaches the stratosphere if the mass flux at the vent is steady. A fluctuating emission rate or an efficient final lift-off due to seawater interaction were unlikely to have increased this loading. Combined with petrological estimates of the degassed S, our results suggest that the eruption emitted 46.5-60.4 Tg S into the troposphere and injected 1.6-15.5 Tg S into the stratosphere, which controlled the atmospheric forcing and the subsequent climate response.
ITRE: Low-light Image Enhancement Based on Illumination Transmission Ratio Estimation

Noise, artifacts, and over-exposure are significant challenges in the field of low-light image enhancement. Existing methods often struggle to address these issues simultaneously. In this paper, we propose a novel Retinex-based method, called ITRE, which suppresses noise and artifacts from the origin of the model, prevents over-exposure throughout the enhancement process. Specifically, we assume that there must exist a pixel which is least disturbed by low light within pixels of same color. First, clustering the pixels on the RGB color space to find the Illumination Transmission Ratio (ITR) matrix of the whole image, which determines that noise is not over-amplified easily. Next, we consider ITR of the image as the initial illumination transmission map to construct a base model for refined transmission map, which prevents artifacts. Additionally, we design an over-exposure module that captures the fundamental characteristics of pixel over-exposure and seamlessly integrate it into the base model. Finally, there is a possibility of weak enhancement when inter-class distance of pixels with same color is too small. To counteract this, we design a Robust-Guard module that safeguards the robustness of the image enhancement process. Extensive experiments demonstrate the effectiveness of our approach in suppressing noise, preventing artifacts, and controlling over-exposure level simultaneously. Our method performs superiority in qualitative and quantitative performance evaluations by comparing with state-of-the-art methods.
Doppler-only Single-scan 3D Vehicle Odometry

We present a novel 3D odometry method that recovers the full motion of a vehicle only from a Doppler-capable range sensor. It leverages the radial velocities measured from the scene, estimating the sensor's velocity from a single scan. The vehicle's 3D motion, defined by its linear and angular velocities, is calculated taking into consideration its kinematic model which provides a constraint between the velocity measured at the sensor frame and the vehicle frame. Experiments carried out prove the viability of our single-sensor method compared to mounting an additional IMU. Our method provides the translation of the sensor, which cannot be reliably determined from an IMU, as well as its rotation. Its short-term accuracy and fast operation (~5ms) make it a proper candidate to supply the initialization to more complex localization algorithms or mapping pipelines. Not only does it reduce the error of the mapper, but it does so at a comparable level of accuracy as an IMU would. All without the need to mount and calibrate an extra sensor on the vehicle.