Miracle-dz/UniFAR-FTUs huggingface dataset:
Model-Based and Graph-Based Priors for Group Testing

The goal of the group testing problem is to identify a set of defective items within a larger set of items, using suitably-designed tests whose outcomes indicate whether any defective item is present. In this paper, we study how the number of tests can be significantly decreased by leveraging the structural dependencies between the items, i.e., by incorporating prior information. To do so, we pursue two different perspectives: (i) As a generalization of the uniform combinatorial prior, we consider the case that the defective set is uniform over a subset of all possible sets of a given size, and study how this impacts the information-theoretic limits on the number of tests for approximate recovery; (ii) As a generalization of the i.i. d. prior, we introduce a new class of priors based on the Ising model, where the associated graph represents interactions between items. We show that this naturally leads to an Integer Quadratic Program decoder, which can be converted to an Integer Linear Program and/or relaxed to a non-integer variant for improved computational complexity, while maintaining strong empirical recovery performance.
How Human is Human Evaluation? Improving the Gold Standard for NLG with Utility Theory

Human ratings are treated as the gold standard in NLG evaluation. The standard protocol is to collect ratings of generated text, average across annotators, and then rank NLG systems by their average scores. However, little consideration has been given as to whether this approach faithfully captures human preferences. In this work, we analyze this standard protocol through the lens of utility theory in economics. We ﬁrst identify the implicit assumptions it makes about annotators and ﬁnd that these assumptions are often violated in practice, in which case annotator ratings become an unfaithful reﬂection of their preferences. The most egregious violations come from us-ing Likert scales, which provably reverse the direction of the true preference in certain cases. We suggest improvements to the standard protocol to make it more theoretically sound, but even in its improved form, it cannot be used to evaluate open-ended tasks like story generation. For the latter, we propose a new evaluation protocol called system-level probabilistic assessment (SPA). In our experiments, we ﬁnd that according to SPA, annotators prefer larger GPT-3 variants to smaller ones – as expected – with all comparisons being statistically significant. In contrast, the standard protocol only yields signiﬁcant results half the time.
ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions

We present ASSET, a neural architecture for automatically modifying an input high-resolution image according to a user's edits on its semantic segmentation map. Our architecture is based on a transformer with a novel attention mechanism. Our key idea is to sparsify the transformer's attention matrix at high resolutions, guided by dense attention extracted at lower image resolutions. While previous attention mechanisms are computationally too expensive for handling high-resolution images or are overly constrained within specific image regions hampering long-range interactions, our novel attention mechanism is both computationally efficient and effective. Our sparsified attention mechanism is able to capture long-range interactions and context, leading to synthesizing interesting phenomena in scenes, such as reflections of landscapes onto water or flora consistent with the rest of the landscape, that were not possible to generate reliably with previous convnets and transformer approaches. We present qualitative and quantitative results, along with user studies, demonstrating the effectiveness of our method.
UNet#: A UNet-like Redesigning Skip Connections for Medical Image Segmentation

As an essential prerequisite for developing a medical intelligent assistant system, medical image segmentation has received extensive research and concentration from the neural network community. A series of UNet-like networks with encoder-decoder architecture has achieved extraordinary success, in which UNet2+ and UNet3+ redesign skip connections, respectively proposing dense skip connection and full-scale skip connection and dramatically improving compared with UNet in medical image segmentation. However, UNet2+ lacks sufficient information explored from the full scale, which will affect the learning of organs' location and boundary. Although UNet3+ can obtain the full-scale aggregation feature map, owing to the small number of neurons in the structure, it does not satisfy the segmentation of tiny objects when the number of samples is small. This paper proposes a novel network structure combining dense skip connections and full-scale skip connections, named UNet-sharp (UNet\#) for its shape similar to symbol \#. The proposed UNet\# can aggregate feature maps of different scales in the decoder sub-network and capture fine-grained details and coarse-grained semantics from the full scale, which benefits learning the exact location and accurately segmenting the boundary of organs or lesions. We perform deep supervision for model pruning to speed up testing and make it possible for the model to run on mobile devices; furthermore, designing two classification-guided modules to reduce false positives achieves more accurate segmentation results. Various experiments of semantic segmentation and instance segmentation on different modalities (EM, CT, MRI) and dimensions (2D, 3D) datasets, including the nuclei, brain tumor, liver, and lung, demonstrate that the proposed method outperforms state-of-the-art models.
Adaptive Few-Shot Learning Algorithm for Rare Sound Event Detection

Sound event detection is to infer the event by understanding the surrounding environmental sounds. Due to the scarcity of rare sound events, it becomes challenging for the well-trained detectors which have learned too much prior knowledge. Meanwhile, few-shot learning methods promise a good generalization abaility when facing a new limited-data task. Recent approaches have achieved promising results in this field. However, these approaches treat each support example independently, ignoring the information of other examples from the whole task. Because of this, most of previous methods are constrained to generate a same feature embedding for all test-time tasks, which is not adaptive to each inputted data. In this work, we propose a novel task-adaptive module which is easy to plant into any metric-based few-shot learning frameworks. The module could identify the task-relevant feature dimension. Incorporating our module improves the performance considerably on two datasets over baseline methods, especially for the transductive propagation network. Such as +6.8% for 5-way 1-shot accuracy on ESC-50, and +5.9% on noiseESC-50. We investigate our approach in the domain-mismatch setting and also achieve better results than previous methods.
Learning Context-Aware Service Representation for Service Recommendation in Workflow Composition

As increasingly more software services have been published onto the Internet, it becomes critical yet highly challenging to recommend suitable services to facilitate scientific workflow composition. This paper proposes a novel Natural Language Processing (NLP)-inspired approach to recommending services throughout a workflow development process, based on incrementally learning latent service representation from workflow provenance. A work-flow composition process is formalized as a step-wise, context-aware service selection procedure, which is mapped to next-word prediction in a natural language sentence generation. Historical service dependencies are extracted from workflow provenance to build and enrich a knowledge graph. Each path in the knowledge graph reflects a scenario in a data analytics experiment, which is analogous to a sentence in a conversation. All paths are thus formalized as composable service sequences and are mined, using various patterns, from the established knowledge graph to construct a corpus. Service embeddings are then learned by applying deep learning model from the NLP field. Extensive experiments on the real-world dataset demonstrate the effectiveness and efficiency of the approach.
An interpretation of the final fully connected layer

In recent years neural networks have achieved state-of-the-art accuracy for various tasks but the the interpretation of the generated outputs still remains difficult. In this work we attempt to provide a method to understand the learnt weights in the final fully connected layer in image classification models. We motivate our method by drawing a connection between the policy gradient objective in RL and supervised learning objective. We suggest that the commonly used cross entropy based supervised learning objective can be regarded as a special case of the policy gradient objective. Using this insight we propose a method to find the most discriminative and confusing parts of an image. Our method does not make any prior assumption about neural network achitecture and has low computational cost. We apply our method on publicly available pre-trained models and report the generated results.
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing

Despite their strong performance on many tasks, pre-trained language models have been shown to struggle on out-of-distribution compositional generalization. Meanwhile, recent work has shown considerable improvements on many NLP tasks from model scaling. Can scaling up model size also improve compositional generalization in semantic parsing? We evaluate encoder-decoder models up to 11B parameters and decoder-only models up to 540B parameters, and compare model scaling curves for three different methods for applying a pre-trained language model to a new task: fine-tuning all parameters, prompt tuning, and in-context learning. We observe that fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional generalization in semantic parsing evaluations. In-context learning has positive scaling curves, but is generally outperformed by much smaller fine-tuned models. Prompt-tuning can outperform fine-tuning, suggesting further potential improvements from scaling as it exhibits a more positive scaling curve. Additionally, we identify several error trends that vary with model scale. For example, larger models are generally better at modeling the syntax of the output space, but are also more prone to certain types of overfitting. Overall, our study highlights limitations of current techniques for effectively leveraging model scale for compositional generalization, while our analysis also suggests promising directions for future work.
After the Dark Ages

After recalling some puzzles in cosmology and brieﬂy reviewing the Friedmann-Lemaître cos-mos a simple uniﬁed model of the “Dark Sector” is described. This model involves a scalar ﬁeld and a pseudo-scalar axion ﬁeld that give rise to Dark Energy in the form of “quintessence” and to “fuzzy” Dark Matter, respectively. Predictions of the model concerning the late-time evolution of the Universe and possible implications for the problem of the observed Matter-Antimatter Asymmetry in the Universe are sketched.
Improved Venting for Flammability Limit Testing Using ASTM E681 Apparatus

Title of Thesis: IMPROVED VENTING FOR FLAMMABILITY LIMIT TESTING USING ASTM E681 APPARATUS Conor G. McCoy, Master of Science, 2016 Directed by: Associate Professor Peter B. Sunderland Department of Fire Protection Engineering The literature on the determination of flammability limits was reviewed and experts on the ASTM E681 standard were interviewed to identify new means of improving the reproducibility of the ASTM E681 test. Venting was identified as a variable of flammability limits not yet addressed. Limitations of the current system for sealing and venting (a rubber stopper) were identified and addressed by the development of a custom burst disc. The burst disc was evaluated for its ability to hold and maintain a vacuum, its ability to vent at pressures of interest, and for its venting phenomena. The burst disc was deemed to be a satisfactory alternative to the rubber stopper and is recommended to be included in the ASTM E681 standard. IMPROVED VENTING FOR FLAMMABILITY LIMIT TESTING USING ASTM E681 APPARATUS
Simple Recurrence Improves Masked Language Models

In this work, we explore whether modeling recurrence into the Transformer architecture can both be beneficial and efficient, by building an extremely simple recurrent module into the Transformer. We compare our model to baselines following the training and evaluation recipe of BERT. Our results confirm that recurrence can indeed improve Transformer models by a consistent margin, without requiring low-level performance optimizations, and while keeping the number of parameters constant. For example, our base model achieves an absolute improvement of 2.1 points averaged across 10 tasks and also demonstrates increased stability in fine-tuning over a range of learning rates.
Frequency–frequency correlations of single-trajectory spectral densities of Gaussian processes

We investigate the stochastic behavior of the single-trajectory spectral density S(ω,T) of several Gaussian stochastic processes, i.e., Brownian motion, the Ornstein–Uhlenbeck process, the Brownian gyrator model and fractional Brownian motion, as a function of the frequency ω and the observation time T . We evaluate in particular the variance and the frequency–frequency correlation of S(ω,T) for different values of ω. We show that these properties exhibit different behaviors for different physical cases and can therefore be used as a sensitive probe discriminating between different kinds of random motion. These results may prove quite useful in the analysis of experimental and numerical data.
Improving language models fine-tuning with representation consistency targets

Fine-tuning contextualized representations 001 learned by pre-trained language models has 002 become a standard practice in the NLP field. 003 However, pre-trained representations are prone 004 to degradation (also known as representation 005 collapse) during fine-tuning, which leads to in-006 stability, sub-optimal performance, and weak 007 generalization. In this paper, we propose a 008 novel fine-tuning method that avoids represen-009 tation collapse during fine-tuning by discourag-010 ing undesirable changes of the representations. 011 We show that our approach matches or exceeds 012 the performance of the existing regularization-013 based fine-tuning methods across 13 language 014 understanding tasks (GLUE benchmark and six 015 additional datasets). We also demonstrate its 016 effectiveness in low-data settings and robust-017 ness to label perturbation. Furthermore, we 018 extend previous studies of representation col-019 lapse and propose several metrics to quantify it. 020 Using these metrics and previously proposed 021 experiments, we show that our approach ob-022 tains significant improvements in retaining the 023 expressive power of representations. 024
PatchNR: Learning from Small Data by Patch Normalizing Flow Regularization

Learning neural networks using only a small amount of data is an important on-going research topic with tremendous potential for applications. In this paper, we introduce a regularizer for the variational modeling of inverse problems in imaging based on normalizing ﬂows. Our regularizer, called patchNR, involves a normalizing ﬂow learned on patches of very few images. The subsequent reconstruction method is completely unsupervised and the same regularizer can be used for different forward operators acting on the same class of images. By investigating the distribution of patches versus those of the whole image class, we prove that our variational model is indeed a MAP approach. Our model can be generalized to conditional patchNRs, if additional supervised information is available. Numerical examples for low-dose CT, limited-angle CT and superresolution of material images demonstrate that our method provides high quality results among unsupervised methods, but requires only few data.
Euclid: Cosmological forecasts from the void size function

The Euclid mission $-$ with its spectroscopic galaxy survey covering a sky area over $15\,000 \ \mathrm{deg}^2$ in the redshift range $0.9<z<1.8\ -$ will provide a sample of tens of thousands of cosmic voids. This paper explores for the first time the constraining power of the void size function on the properties of dark energy (DE) from a survey mock catalogue, the official Euclid Flagship simulation. We identify voids in the Flagship light-cone, which closely matches the features of the upcoming Euclid spectroscopic data set. We model the void size function considering a state-of-the art methodology: we rely on the volume conserving (Vdn) model, a modification of the popular Sheth&van de Weygaert model for void number counts, extended by means of a linear function of the large-scale galaxy bias. We find an excellent agreement between model predictions and measured mock void number counts. We compute updated forecasts for the Euclid mission on DE from the void size function and provide reliable void number estimates to serve as a basis for further forecasts of cosmological applications using voids. We analyse two different cosmological models for DE: the first described by a constant DE equation of state parameter, $w$, and the second by a dynamic equation of state with coefficients $w_0$ and $w_a$. We forecast $1\sigma$ errors on $w$ lower than $10\%$, and we estimate an expected figure of merit (FoM) for the dynamical DE scenario $\mathrm{FoM}_{w_0,w_a} = 17$ when considering only the neutrino mass as additional free parameter of the model. The analysis is based on conservative assumptions to ensure full robustness, and is a pathfinder for future enhancements of the technique. Our results showcase the impressive constraining power of the void size function from the Euclid spectroscopic sample, both as a stand-alone probe, and to be combined with other Euclid cosmological probes.
CDFKD-MFS: Collaborative Data-Free Knowledge Distillation via Multi-Level Feature Sharing

Recently, the compression and deployment of powerful deep neural networks (DNNs) on resource-limited edge devices to provide intelligent services have become attractive tasks. Although knowledge distillation (KD) is a feasible solution for compression, its requirement on the original dataset raises privacy concerns. In addition, it is common to integrate multiple pretrained models to achieve satisfactory performance. How to compress multiple models into a tiny model is challenging, especially when the original data are unavailable. To tackle this challenge, we propose a framework termed collaborative data-free knowledge distillation via multi-level feature sharing (CDFKD-MFS), which consists of a multi-header student module, an asymmetric adversarial data-free KD module, and an attention-based aggregation module. In this framework, the student model equipped with a multi-level feature-sharing structure learns from multiple teacher models and is trained together with a generator in an asymmetric adversarial manner. When some real samples are available, the attention module adaptively aggregates predictions of the student headers, which can further improve performance. We conduct extensive experiments on three popular computer visual datasets. In particular, compared with the most competitive alternative, the accuracy of the proposed framework is 1.18% higher on the CIFAR-100 dataset, 1.67% higher on the Caltech-101 dataset, and 2.99% higher on the mini-ImageNet dataset.
Compression-aware Training of Neural Networks using Frank-Wolfe

Many existing Neural Network pruning approaches rely on either retraining or inducing a strong bias in order to converge to a sparse solution throughout training. A third paradigm, 'compression-aware' training, aims to obtain state-of-the-art dense models that are robust to a wide range of compression ratios using a single dense training run while also avoiding retraining. We propose a framework centered around a versatile family of norm constraints and the Stochastic Frank-Wolfe (SFW) algorithm that encourage convergence to well-performing solutions while inducing robustness towards convolutional filter pruning and low-rank matrix decomposition. Our method is able to outperform existing compression-aware approaches and, in the case of low-rank matrix decomposition, it also requires significantly less computational resources than approaches based on nuclear-norm regularization. Our findings indicate that dynamically adjusting the learning rate of SFW, as suggested by Pokutta et al. (2020), is crucial for convergence and robustness of SFW-trained models and we establish a theoretical foundation for that practice.
Multiring images of thin accretion disk of a regular naked compact object

(Dated: We discuss the importance of multi-ring images in the optical appearance of a horizonless spherically symmetric compact object, when illuminated by an optically thin accretion disk. Such an object corresponds to a sub-case of an analytically tractable extension of the Kerr solution dubbed as the eye of the storm by Simpson and Visser in [JCAP 03 (2022) 011], which merits in removing curvature singularities via an asymptotically Minkowski core, while harbouring both a critical curve and an inﬁnite potential barrier at the center for null geodesics. This multi-ring structure is induced by light rays winding several times around the object, and whose luminosity is signiﬁcantly boosted as compared to the Schwarzschild solution by the modiﬁed shape of the potential. Using three toy-proﬁles for the emission of an inﬁnitely thin disk, truncated at its inner edge (taking its maximum value there) and having diﬀerent decays with the distance, we discuss the image created by up to eight rings superimposed on top of the direct emission of the disk as its edge is moved closer to the center of the object. Our results point out to the existence of multi-ring images with a non-negligible luminosity in shadow observations when one allows for the existence of other compact objects in the cosmic zoo beyond the Schwarzschild solution. Such multi-ring images could be detectable within the future projects on very-long baseline interferometry.
Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models

The emergent cross-lingual transfer seen in multilingual pretrained models has sparked significant interest in studying their behavior. However, because these analyses have focused on fully trained multilingual models, little is known about the dynamics of the multilingual pretraining process. We investigate when these models acquire their in-language and cross-lingual abilities by probing checkpoints taken from throughout XLM-R pretraining, using a suite of linguistic tasks. Our analysis shows that the model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones. In contrast, the point in pretraining when the model learns to transfer cross-lingually differs across language pairs. Interestingly, we also observe that, across many languages and tasks, the final model layer exhibits significant performance degradation over time, while linguistic knowledge propagates to lower layers of the network. Taken together, these insights highlight the complexity of multilingual pretraining and the resulting varied behavior for different languages over time.