Miracle-dz/UniFAR-FTUs huggingface dataset:
DISeR: Designing Imaging Systems with Reinforcement Learning

Imaging systems consist of cameras to encode visual information about the world and perception models to interpret this encoding. Cameras contain (1) illumination sources, (2) optical elements, and (3) sensors, while perception models use (4) algorithms. Directly searching over all combinations of these four building blocks to design an imaging system is challenging due to the size of the search space. Moreover, cameras and perception models are often designed independently, leading to sub-optimal task performance. In this paper, we formulate these four building blocks of imaging systems as a context-free grammar (CFG), which can be automatically searched over with a learned camera designer to jointly optimize the imaging system with task-specific perception models. By transforming the CFG to a state-action space, we then show how the camera designer can be implemented with reinforcement learning to intelligently search over the combinatorial space of possible imaging system configurations. We demonstrate our approach on two tasks, depth estimation and camera rig design for autonomous vehicles, showing that our method yields rigs that outperform industry-wide standards. We believe that our proposed approach is an important step towards automating imaging system design. Our project page is https://tzofi.github. io/diser.
BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLP

Large Language Models (LLMs) have emerged as one of the most important breakthroughs in NLP for their impressive skills in language generation and other language-specific tasks. Though LLMs have been evaluated in various tasks, mostly in English, they have not yet undergone thorough evaluation in under-resourced languages such as Bengali (Bangla). To this end, this paper introduces BenLLM-Eval, which consists of a comprehensive evaluation of LLMs to benchmark their performance in the Bengali language that has modest resources. In this regard, we select various important and diverse Bengali NLP tasks, such as text summarization, question answering, paraphrasing, natural language inference, transliteration, text classification, and sentiment analysis for zero-shot evaluation of popular LLMs, namely, GPT-3.5, LLaMA-2-13b-chat, and Claude-2. Our experimental results demonstrate that while in some Bengali NLP tasks, zero-shot LLMs could achieve performance on par, or even better than current SOTA fine-tuned models; in most tasks, their performance is quite poor (with the performance of open-source LLMs like LLaMA-2-13b-chat being significantly bad) in comparison to the current SOTA results. Therefore, it calls for further efforts to develop a better understanding of LLMs in modest-resourced languages like Bengali.
Multi-Dimensional Hyena for Spatial Inductive Bias

In recent years, Vision Transformers have attracted increasing interest from computer vision researchers. However, the advantage of these transformers over CNNs is only fully manifested when trained over a large dataset, mainly due to the reduced inductive bias towards spatial locality within the transformer's self-attention mechanism. In this work, we present a data-efficient vision transformer that does not rely on self-attention. Instead, it employs a novel generalization to multiple axes of the very recent Hyena layer. We propose several alternative approaches for obtaining this generalization and delve into their unique distinctions and considerations from both empirical and theoretical perspectives. Our empirical findings indicate that the proposed Hyena N-D layer boosts the performance of various Vision Transformer architectures, such as ViT, Swin, and DeiT across multiple datasets. Furthermore, in the small dataset regime, our Hyena-based ViT is favorable to ViT variants from the recent literature that are specifically designed for solving the same challenge, i.e., working with small datasets or incorporating image-specific inductive bias into the self-attention mechanism. Finally, we show that a hybrid approach that is based on Hyena N-D for the first layers in ViT, followed by layers that incorporate conventional attention, consistently boosts the performance of various vision transformer architectures.
VidChapters-7M: Video Chapters at Scale

Segmenting long videos into chapters enables users to quickly navigate to the information of their interest. This important topic has been understudied due to the lack of publicly released datasets. To address this issue, we present VidChapters-7M, a dataset of 817K user-chaptered videos including 7M chapters in total. VidChapters-7M is automatically created from videos online in a scalable manner by scraping user-annotated chapters and hence without any additional manual annotation. We introduce the following three tasks based on this data. First, the video chapter generation task consists of temporally segmenting the video and generating a chapter title for each segment. To further dissect the problem, we also define two variants of this task: video chapter generation given ground-truth boundaries, which requires generating a chapter title given an annotated video segment, and video chapter grounding, which requires temporally localizing a chapter given its annotated title. We benchmark both simple baselines and state-of-the-art video-language models for these three tasks. We also show that pretraining on VidChapters-7M transfers well to dense video captioning tasks in both zero-shot and finetuning settings, largely improving the state of the art on the YouCook2 and ViTT benchmarks. Finally, our experiments reveal that downstream performance scales well with the size of the pretraining dataset. Our dataset, code, and models are publicly available at https://antoyang.github. io/vidchapters.html.
Modern Variation and Evolutionary Change in the Hominin Eye Orbit

The principal aim of this research is to investigate variation and change in orbital morphology among past and modern human groups. This involves a comparison of individuals from Western Europe (68), China (57), and South Africa (54); an investigation of how the orbit varies amid the neurocranium and lower face in Pan, Australopithecus africanus, Homo erectus, Archaic Homo sapiens, and anatomically modern Homo sapiens representing different grades of cranial expansion and reduced facial prognathism; an analysis of more recent change in orbital morphology among Western European groups dating to the Upper Paleolithic; as well as an examination of how temporal and spatial variation in orbital and overall craniofacial form may relate to a recent global trend of reduced visual acuity in humans. Univariate and multivariate statistical tools are used to test hypotheses relating to these aspects of variation and evolutionary change. Analysis of Variance (ANOVA) reveals many differences in orbital form among modern human groups, with the greatest degree of inter-group variation observed for orbital volume, orbital depth, and shape of the orbital margins, while no difference was found for interorbital breadth, and biorbital breadth. Mahalanobis distance and canonical discriminant function analyses indicate that despite these differences, the orbits are less variable compared to broader traits of the skull, and that groups with a similar overall craniofacial form can differ markedly in orbital morphology. The orbits are found to vary in association with a grade shift in cranial size and facial prognathism, becoming more retracted relative to basion, more rectangular, and more frontated in association with increased cranial size and decreased lower facial projection. During the last 30,000 years in Western Europe, the orbits have shifted posteriorly relative to basion, while internal orbital depth has also been reduced. Shape of the orbital margins have changed most throughout this period, becoming taller and narrower, while orbital volume, orbital frontation, and interorbital breadth show no relationship to time. An examination of eyeball size, orbit size, and spherical equivalent refractive error (SER) reveals a strong relationship between relative size of the eye within the orbit and the incidence/severity of myopia. This widespread neutral human trait is examined in the context of variation in orbital and overall craniofacial form between the sexes, among modern human populations, and throughout hominin evolution. These results indicate that future research into the etiology of juvenile-onset myopia should consider how the eyeball interacts with the matrix of structural and functional components of the skull during ontogenetic and evolutionary morphogenesis.
Numerical and Experimental Investigation of Tandem Wing Flyers

The recent focus on micro-UAV systems and bio-inspired drones has generated interest in tandem wing applications. Dragonfly-based configurations are of significance for very low Reynolds numbers; for larger drones, Microraptor-based geometries could prove to be efficient. The present study of tandem wing flyers aims at understanding the basic principles governing the aerodynamic properties of tandem wings in close proximity. The analysis includes both numerical simulations by means of the Unsteady Vortex Lattice Method and wind tunnel experimentation applied to generic rectangular wing geometries. Preliminary conclusions include the facts that increasing the rear wing’s angle of attack results in a bigger increase in lift than increasing the front wing’s angle of attack. The dihedral angles of the two wings also seem to have significant impact on the lift, some configurations leading to an increase in lift coefficient of up to 25%. The insight provided by the results will be used in the future to test and validate different flight configurations for the Microraptor and, hopefully, to shed some light on its preferred in-flight configuration and its flight capabilities.
Deep Learning with Photonic Neural Cellular Automata

Rapid advancements in deep learning over the past decade have fueled an insatiable demand for efficient and scalable hardware. Photonics offers a promising solution by leveraging the unique properties of light. However, conventional neural network architectures, which typically require dense programmable connections, pose several practical challenges for photonic realizations. To overcome these limitations, we propose and experimentally demonstrate Photonic Neural Cellular Automata (PNCA) for photonic deep learning with sparse connectivity. PNCA harnesses the speed and interconnectivity of photonics, as well as the self-organizing nature of cellular automata through local interactions to achieve robust, reliable, and efficient processing. We utilize linear light interference and parametric nonlinear optics for all-optical computations in a time-multiplexed photonic network to experimentally perform self-organized image classification. We demonstrate binary classification of images in the fashion-MNIST dataset using as few as 3 programmable photonic parameters, achieving an experimental accuracy of 98.0% with the ability to also recognize out-of-distribution data. The proposed PNCA approach can be adapted to a wide range of existing photonic hardware and provides a compelling alternative to conventional photonic neural networks by maximizing the advantages of light-based computing whilst mitigating their practical challenges. Our results showcase the potential of PNCA in advancing photonic deep learning and highlights a path for next-generation photonic computers.
C2VAE: Gaussian Copula-based VAE Differing Disentangled from Coupled Representations with Contrastive Posterior

We present a self-supervised variational autoencoder (VAE) to jointly learn disentangled and dependent hidden factors and then enhance disentangled representation learning by a self-supervised classifier to eliminate coupled representations in a contrastive manner. To this end, a Contrastive Copula VAE (C$^2$VAE) is introduced without relying on prior knowledge about data in the probabilistic principle and involving strong modeling assumptions on the posterior in the neural architecture. C$^2$VAE simultaneously factorizes the posterior (evidence lower bound, ELBO) with total correlation (TC)-driven decomposition for learning factorized disentangled representations and extracts the dependencies between hidden features by a neural Gaussian copula for copula coupled representations. Then, a self-supervised contrastive classifier differentiates the disentangled representations from the coupled representations, where a contrastive loss regularizes this contrastive classification together with the TC loss for eliminating entangled factors and strengthening disentangled representations. C$^2$VAE demonstrates a strong effect in enhancing disentangled representation learning. C$^2$VAE further contributes to improved optimization addressing the TC-based VAE instability and the trade-off between reconstruction and representation.
Probabilistic Weight Fixing: Large-scale training of neural network weight uncertainties for quantization

Weight-sharing quantization has emerged as a technique to reduce energy expenditure during inference in large neural networks by constraining their weights to a limited set of values. However, existing methods for weight-sharing quantization often make assumptions about the treatment of weights based on value alone that neglect the unique role weight position plays. This paper proposes a probabilistic framework based on Bayesian neural networks (BNNs) and a variational relaxation to identify which weights can be moved to which cluster centre and to what degree based on their individual position-specific learned uncertainty distributions. We introduce a new initialisation setting and a regularisation term which allow for the training of BNNs under complex dataset-model combinations. By leveraging the flexibility of weight values captured through a probability distribution, we enhance noise resilience and downstream compressibility. Our iterative clustering procedure demonstrates superior compressibility and higher accuracy compared to state-of-the-art methods on both ResNet models and the more complex transformer-based architectures. In particular, our method outperforms the state-of-the-art quantization method top-1 accuracy by 1.6% on ImageNet using DeiT-Tiny, with its 5 million+ weights now represented by only 296 unique values.
Policy Stitching: Learning Transferable Robot Policies

Training robots with reinforcement learning (RL) typically involves heavy interactions with the environment, and the acquired skills are often sensitive to changes in task environments and robot kinematics. Transfer RL aims to leverage previous knowledge to accelerate learning of new tasks or new body configurations. However, existing methods struggle to generalize to novel robot-task combinations and scale to realistic tasks due to complex architecture design or strong regularization that limits the capacity of the learned policy. We propose Policy Stitching, a novel framework that facilitates robot transfer learning for novel combinations of robots and tasks. Our key idea is to apply modular policy design and align the latent representations between the modular interfaces. Our method allows direct stitching of the robot and task modules trained separately to form a new policy for fast adaptation. Our simulated and real-world experiments on various 3D manipulation tasks demonstrate the superior zero-shot and few-shot transfer learning performances of our method. Our project website is at: http://generalroboticslab.com/PolicyStitching/ .
A new analytical model of magnetofluids surrounding rotating black holes

In this study, we develop a simplified magnetofluid model in the framework of GRMHD. We consider an ideal, adiabatic fluid composed of two components, ions and electrons, having a constant ratio between their temperatures. The flows are assumed to be governed by gravity, enabling us to employ the ballistic approximation, treating the streamlines as timelike geodesics. We show that the model is analytically solvable around a rotating black hole if the angular velocity of the geodesic uθ is vanishing. In the corresponding solution, which is named the conical solution, we derive a comprehensive set of explicit expressions for the thermodynamics and the associated magnetic field. Furthermore, we explore the potential applications of our model to describe the thick disks and the jets at the horizon scale. Our model provides a direct pathway for the study of black hole imaging.
PRiSM: Enhancing Low-Resource Document-Level Relation Extraction with Relation-Aware Score Calibration

Document-level relation extraction (DocRE) aims to extract relations of all entity pairs in a document. A key challenge in DocRE is the cost of annotating such data which requires intensive human effort. Thus, we investigate the case of DocRE in a low-resource setting, and we find that existing models trained on low data overestimate the NA ("no relation") label, causing limited performance. In this work, we approach the problem from a calibration perspective and propose PRiSM, which learns to adapt logits based on relation semantic information. We evaluate our method on three DocRE datasets and demonstrate that integrating existing models with PRiSM improves performance by as much as 26.38 F1 score, while the calibration error drops as much as 36 times when trained with about 3% of data. The code is publicly available at https://github.com/brightjade/PRiSM.
ReMasker: Imputing Tabular Data with Masked Autoencoding

We present ReMasker, a new method of imputing missing values in tabular data by extending the masked autoencoding framework. Compared with prior work, ReMasker is both simple -- besides the missing values (i.e., naturally masked), we randomly ``re-mask'' another set of values, optimize the autoencoder by reconstructing this re-masked set, and apply the trained model to predict the missing values; and effective -- with extensive evaluation on benchmark datasets, we show that ReMasker performs on par with or outperforms state-of-the-art methods in terms of both imputation fidelity and utility under various missingness settings, while its performance advantage often increases with the ratio of missing data. We further explore theoretical justification for its effectiveness, showing that ReMasker tends to learn missingness-invariant representations of tabular data. Our findings indicate that masked modeling represents a promising direction for further research on tabular data imputation. The code is publicly available.
Near-field Hybrid Beamforming for Terahertz-band Integrated Sensing and Communications

Terahertz (THz) band communications and integrated sensing and communications (ISAC) are two main facets of the sixth generation wireless networks. In order to compensate the severe attenuation, the THz wireless systems employ large arrays, wherein the near-field beam-squint severely degrades the beamforming accuracy. Contrary to prior works that examine only either narrowband ISAC beamforming or far-field models, we introduce an alternating optimization technique for hybrid beamforming design in near-field THz-ISAC scenario. We also propose an efficient approach to compensate near-field beam-squint via baseband beamformers. Via numerical simulations, we show that the proposed approach achieves satisfactory spectral efficiency performance while accurately estimating the near-field beam- formers and mitigating the beam-squint without additional hardware components.
Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models

Because diffusion models have shown impressive performances in a number of tasks, such as image synthesis, there is a trend in recent works to prove (with certain assumptions) that these models have strong approximation capabilities. In this paper, we show that current diffusion models actually have an expressive bottleneck in backward denoising and some assumption made by existing theoretical guarantees is too strong. Based on this finding, we prove that diffusion models have unbounded errors in both local and global denoising. In light of our theoretical studies, we introduce soft mixture denoising (SMD), an expressive and efficient model for backward denoising. SMD not only permits diffusion models to well approximate any Gaussian mixture distributions in theory, but also is simple and efficient for implementation. Our experiments on multiple image datasets show that SMD significantly improves different types of diffusion models (e.g., DDPM), espeically in the situation of few backward iterations.
Masked Discriminators for Content-Consistent Unpaired Image-to-Image Translation

A common goal of unpaired image-to-image translation is to preserve content consistency between source images and translated images while mimicking the style of the target domain. Due to biases between the datasets of both domains, many methods suffer from inconsistencies caused by the translation process. Most approaches introduced to mitigate these inconsistencies do not constrain the discriminator, leading to an even more ill-posed training setup. Moreover, none of these approaches is designed for larger crop sizes. In this work, we show that masking the inputs of a global discriminator for both domains with a content-based mask is sufficient to reduce content inconsistencies significantly. However, this strategy leads to artifacts that can be traced back to the masking process. To reduce these artifacts, we introduce a local discriminator that operates on pairs of small crops selected with a similarity sampling strategy. Furthermore, we apply this sampling strategy to sample global input crops from the source and target dataset. In addition, we propose feature-attentive denormalization to selectively incorporate content-based statistics into the generator stream. In our experiments, we show that our method achieves state-of-the-art performance in photorealistic sim-to-real translation and weather translation and also performs well in day-to-night translation. Additionally, we propose the cKVD metric, which builds on the sKVD metric and enables the examination of translation quality at the class or category level.
Reservoir Geomechanics: Index

This interdisciplinary book encompasses the fields of rock mechanics, structural geology, and petroleum engineering to address a wide range of geomechanical problems that arise during the exploitation of oil and gas reservoirs. Covering the exploration, assessment, and production phases of petroleum reservoir development, the book considers key practical issues such as prediction of pore pressure; estimation of hydrocarbon column heights and fault seal potential; determination of optimally stable well trajectories; casing set points and mud weights; changes in reservoir performance during depletion; and production-induced faulting and subsidence. The first part of the book establishes the basic principles of geomechanics in a way that allows readers from different disciplinary backgrounds to understand the key concepts. It then goes on to introduce practical measurement and experimental techniques before illustrating their successful application, through case studies taken from oil and gas fields around the world, to improve recovery and reduce exploitation costs. Reservoir Geomechanics is a practical reference for geoscientists and engineers in the petroleum and geothermal industries, and for research scientists interested in stress measurements and their application to problems of faulting and fluid flow in the crust.