Miracle-dz/UniFAR-FTUs huggingface dataset:
Shedding New Light on the Language of the Dark Web

The hidden nature and the limited accessibility of the Dark Web, combined with the lack of public datasets in this domain, make it difficult to study its inherent characteristics such as linguistic properties. Previous works on text classification of Dark Web domain have suggested that the use of deep neural models may be ineffective, potentially due to the linguistic differences between the Dark and Surface Webs. However, not much work has been done to uncover the linguistic characteristics of the Dark Web. This paper introduces CoDA, a publicly available Dark Web dataset consisting of 10000 web documents tailored towards text-based Dark Web analysis. By leveraging CoDA, we conduct a thorough linguistic analysis of the Dark Web and examine the textual differences between the Dark Web and the Surface Web. We also assess the performance of various methods of Dark Web page classification. Finally, we compare CoDA with an existing public Dark Web dataset and evaluate their suitability for various use cases.
Deep Vehicle Detection in Satellite Video

This work presents a deep learning approach for vehicle detection in satellite video. Vehicle detection is perhaps impossible in single EO satellite images due to the tininess of vehicles (4-10 pixel) and their similarity to the background. Instead, we consider satellite video which overcomes the lack of spatial information by temporal consistency of vehicle movement. A new spatiotemporal model of a compact $3 \times 3$ convolutional, neural network is proposed which neglects pooling layers and uses leaky ReLUs. Then we use a reformulation of the output heatmap including Non-Maximum-Suppression (NMS) for the final segmentation. Empirical results on two new annotated satellite videos reconfirm the applicability of this approach for vehicle detection. They more importantly indicate that pre-training on WAMI data and then fine-tuning on few annotated video frames for a new video is sufficient. In our experiment only five annotated images yield a $F_1$ score of 0.81 on a new video showing more complex traffic patterns than the Las Vegas video. Our best result on Las Vegas is a $F_1$ score of 0.87 which makes the proposed approach a leading method for this benchmark.
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals

We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. Originated in ELECTRA, this training strategy has demonstrated sample-efficiency to pretrain models at the scale of hundreds of millions of parameters. In this work, we conduct a comprehensive empirical study, and propose a recipe, namely"Model generated dEnoising TRaining Objective"(METRO), which incorporates some of the best modeling techniques developed recently to speed up, stabilize, and enhance pretrained language models without compromising model effectiveness. The resultant models, METRO-LM, consisting of up to 5.4 billion parameters, achieve new state-of-the-art on the GLUE, SuperGLUE, and SQuAD benchmarks. More importantly, METRO-LM are efficient in that they often outperform previous large models with significantly smaller model sizes and lower pretraining cost.
GIFS: Neural Implicit Function for General Shape Representation

Recent development of neural implicit function has shown tremendous success on high-quality 3D shape re-construction. However, most works divide the space into inside and outside of the shape, which limits their repre-senting power to single-layer and watertight shapes. This limitation leads to tedious data processing (converting non-watertight raw data to watertight) as well as the incapability of representing general object shapes in the real world. In this work, we propose a novel method to represent general shapes including non-watertight shapes and shapes with multi-layer surfaces. We introduce General Implicit Function for 3D Shape (GIFS), which models the relationships between every two points instead of the relationships between points and surfaces. Instead of dividing 3D space into predefined inside-outside regions, GIFS encodes whether two points are separated by any surface. Experiments on ShapeNet show that GIFS outperforms previous state-of-the-art methods in terms of reconstruction quality, rendering efficiency, and visual fidelity. Project page is available at https://jianglongye.com/gifs.
Stellar Halos from the The Dragonfly Edge-on Galaxies Survey

We present the primary results from the Dragonfly Edge-on Galaxies Survey, an exploration of the stellar halos of twelve nearby (d < 25 Mpc) edge-on disk galaxies with the Dragonfly Telephoto Array. The edge-on orientation of these galaxies allows their stellar halos to be explored with minimal obscuration by or confusion with the much brighter disk light. Galaxies in the sample span a range of stellar masses from 109.68 to 1010.88 M ⊙. We confirm that the wide range of stellar halo mass fractions previously seen for Milky Way–mass galaxies is also found among less massive spiral galaxies. The scatter in stellar halo mass fraction is large, but we do find a significant positive correlation between stellar halo mass fraction and total stellar mass when the former is measured beyond five half-mass radii. Reasonably good agreement is found with predictions from cosmological hydrodynamical simulations, although observed stellar halo fractions appear to be somewhat lower than expected from these simulations.
Semi-Supervised Convolutive NMF for Automatic Piano Transcription

Automatic Music Transcription, which consists in transforming an audio recording of a musical performance into symbolic format, remains a difficult Music Information Retrieval task. In this work, which focuses on piano transcription, we propose a semi-supervised approach using low-rank matrix factorization techniques, in particular Convolutive Nonnegative Matrix Factorization. In the semi-supervised setting, only a single recording of each individual notes is required. We show on the MAPS dataset that the proposed semi-supervised CNMF method performs better than state-of-the-art low-rank factorization techniques and a little worse than supervised deep learning state-of-the-art methods, while however suffering from generalization issues.
Boundary-bulk patterning in three-dimensional conﬂuent cellular collectives

Organoids are in vitro cellular collectives from which brain-like, or gut-like, or kidney-like structures emerge. To make quantitative predictions regarding the morphology and rheology of a cellular collective in its initial stages of development, we construct and study a three-dimensional vertex model. In such a model, the cells are represented as deformable polyhedrons with cells sharing faces such that there are no gaps between them, otherwise known as conﬂuent. In a bulk model with periodic boundary conditions, we ﬁnd a rigidity transition as a function of the target cell shape index s 0 with a critical value s ∗ 0 = 5 . 39 ± 0 . 01. For a conﬂuent cellular collective with a ﬁnite boundary, and in the presence of lateral extensile and in-plane, radial extensile deformations, we ﬁnd signiﬁcant boundary-bulk patterning that is one-cell layer thick. More speciﬁcally, for lateral extensile deformations, the cells in the bulk are much less aligned with the direction of the lateral deformation than the cells at the boundary. For in-plane, radial deformations, the cells in the bulk exhibit much less reorientation perpendicular to the radial direction than the cells at the boundary. In other words, for both deformations, the bulk cells are insulated from the deformations, at least over time scales much slower than the timescale for cellular rearrangements. Our results provide an underlying mechanism for some observed cell shape patterning in organoids. Fi-nally, we discuss the use of a cellular-based approach to designing organoids with new types of morphologies to study the intricate relationship between structure and function at the multi-cellular scale.
Uplifting Interviews in Social Science with Individual Data Visualization: the case of Music Listening

Collecting accurate and fine-grain information about the music people like, dislike and actually listen to has long been a challenge for sociologists. As millions of people now use online music streaming services, research can build upon the individual listening history data that are collected by these platforms. Individual interviews, in particular, can benefit from such data, by allowing the interviewers to immerse themselves in the musical universe of consenting respondents, and thus ask them contextualized questions and get more precise answers. Designing a visual exploration tool allowing such an immersion is however difficult, because of the volume and heterogeneity of the listening data, the unequal “visual literacy” of the prospective users, or the interviewers’ potential lack of knowledge of the music listened to by the respondents. In this case study we discuss the design and evaluation of such a tool. Designed with social scientists, its purpose is to help them in preparing and conducting semi-structured interviews that address various aspects of the listening experience. It was evaluated during thirty interviews with consenting users of a streaming platform in France.
Towards Metrical Reconstruction of Human Faces

Face reconstruction and tracking is a building block of numerous applications in AR/VR, human-machine interac-tion, as well as medical applications. Most of these applications rely on a metrically correct prediction of the shape, especially, when the reconstructed subject is put into a metrical context (i.e., when there is a reference object of known size). A metrical reconstruction is also needed for any application that measures distances and dimensions of the subject (e.g., to virtually ﬁt a glasses frame). State-of-the-art methods for face reconstruction from a single image are trained on large 2D image datasets in a self-supervised fashion. However, due to the nature of a perspective projection they are not able to reconstruct the actual face dimensions, and even predicting the average human face outperforms some of these methods in a metrical sense. To learn the actual shape of a face, we argue for a supervised training scheme. Since there exists no large-scale 3D dataset for this task, we annotated and uniﬁed small- and medium-scale databases. The resulting uniﬁed dataset is still a medium-scale dataset with more than 2k identities and training purely on it would lead to overﬁtting. To this end, we take advantage of a face recognition network pretrained on a large-scale 2D image dataset, which provides distinct features for different faces and is robust to expression, illumination, and camera changes. Using these features, we train our face shape estimator in a supervised fashion, inheriting the robustness and generalization of the face recognition network. Our method, which we call MICA (MetrIC fAce), outperforms the state-of-the-art reconstruction methods by a large margin, both on current non-metric benchmarks as well as on our metric benchmarks ( 15% and 24% lower average error on NoW, respec-tively). Project website: https://zielon.github. io/mica/.
Constraining Nucleosynthesis in Neutrino-driven Winds: Observations, Simulations, and Nuclear Physics

A promising astrophysical site to produce the lighter heavy elements of the first r-process peak (Z = 38 − 47) is the moderately neutron-rich (0.4 < Y e < 0.5) neutrino-driven ejecta of explosive environments, such as core-collapse supernovae and neutron star mergers, where the weak r-process operates. This nucleosynthesis exhibits uncertainties from the absence of experimental data from (α, xn) reactions on neutron-rich nuclei, which are currently based on statistical model estimates. In this work, we report on a new study of the nuclear reaction impact using a Monte Carlo approach and improved (α, xn) rates based on the Atomki-V2 α optical model potential. We compare our results with observations from an up-to-date list of metal-poor stars with [Fe/H] < −1.5 to find conditions of the neutrino-driven wind where the lighter heavy elements can be synthesized. We identified a list of (α, xn) reaction rates that affect key elemental ratios in different astrophysical conditions. Our study aims to motivate more nuclear physics experiments on (α, xn) reactions using the current and new generation of radioactive beam facilities and also more observational studies of metal-poor stars.
Self-Guided Learning to Denoise for Robust Recommendation

The ubiquity of implicit feedback makes them the default choice to build modern recommender systems. Generally speaking, observed interactions are considered as positive samples, while unobserved interactions are considered as negative ones. However, implicit feedback is inherently noisy because of the ubiquitous presence of noisy-positive and noisy-negative interactions. Recently, some studies have noticed the importance of denoising implicit feedback for recommendations, and enhanced the robustness of recommendation models to some extent. Nonetheless, they typically fail to (1) capture the hard yet clean interactions for learning comprehensive user preference, and (2) provide a universal denoising solution that can be applied to various kinds of recommendation models. In this paper, we thoroughly investigate the memorization effect of recommendation models, and propose a new denoising paradigm, i.e., Self-Guided Denoising Learning (SGDL), which is able to collect memorized interactions at the early stage of the training (i.e., ''noise-resistant'' period), and leverage those data as denoising signals to guide the following training (i.e., ''noise-sensitive'' period) of the model in a meta-learning manner. Besides, our method can automatically switch its learning phase at the memorization point from memorization to self-guided learning, and select clean and informative memorized data via a novel adaptive denoising scheduler to improve the robustness. We incorporate SGDL with four representative recommendation models (i.e., NeuMF, CDAE, NGCF and LightGCN) and different loss functions (i.e., binary cross-entropy and BPR loss). The experimental results on three benchmark datasets demonstrate the effectiveness of SGDL over the state-of-the-art denoising methods like T-CE, IR, DeCA, and even state-of-the-art robust graph-based methods like SGCN and SGL.
Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity

Abstract This paper analyzes three formal models of Transformer encoders that differ in the form of their self-attention mechanism: unique hard attention (UHAT); generalized unique hard attention (GUHAT), which generalizes UHAT; and averaging hard attention (AHAT). We show that UHAT and GUHAT Transformers, viewed as string acceptors, can only recognize formal languages in the complexity class AC0, the class of languages recognizable by families of Boolean circuits of constant depth and polynomial size. This upper bound subsumes Hahn’s (2020) results that GUHAT cannot recognize the DYCK languages or the PARITY language, since those languages are outside AC0 (Furst et al., 1984). In contrast, the non-AC0 languages MAJORITY and DYCK-1 are recognizable by AHAT networks, implying that AHAT can recognize languages that UHAT and GUHAT cannot.
Clothes-Changing Person Re-identification with RGB Modality Only

The key to address clothes-changing person re-identification (re-id) is to extract clothes-irrelevant features, e.g., face, hairstyle, body shape, and gait. Most current works mainly focus on modeling body shape from multi-modality information (e.g., silhouettes and sketches), but do not make full use of the clothes-irrelevant information in the original RGB images. In this paper, we propose a Clothes-based Adversarial Loss (CAL) to mine clothes-irrelevant features from the original RGB images by penalizing the predictive power of re-id model w.r. t. clothes. Extensive experiments demonstrate that using RGB images only, CAL outperforms all state-of-the-art methods on widely-used clothes-changing person re-id benchmarks. Besides, compared with images, videos contain richer appearance and additional temporal information, which can be used to model proper spatiotemporal patterns to assist clothes-changing re-id. Since there is no publicly available clothes-changing video re-id dataset, we contribute a new dataset named CCVID and show that there exists much room for improvement in modeling spatiotemporal information. The code and new dataset are available at: h t t $p$ s: //github.com/guxinqian/Simple-CCReID.
Revisiting Transformer-based Models for Long Document Classification

The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph documents are common and they cannot be efficiently encoded by vanilla Transformer-based models. We compare different Transformer-based Long Document Classification (TrLDC) approaches that aim to mitigate the computational overhead of vanilla transformers to encode much longer text, namely sparse attention and hierarchical encoding methods. We examine several aspects of sparse attention (e.g., size of local attention window, use of global attention) and hierarchical (e.g., document splitting strategy) transformers on four document classification datasets covering different domains. We observe a clear benefit from being able to process longer text, and, based on our results, we derive practical advice of applying Transformer-based models on long document classification tasks.
Label Semantic Aware Pre-training for Few-shot Text Classification

In text classification tasks, useful information is encoded in the label names. Label semantic aware systems have leveraged this information for improved text classification performance during fine-tuning and prediction. However, use of label-semantics during pre-training has not been extensively explored. We therefore propose Label Semantic Aware Pre-training (LSAP) to improve the generalization and data efficiency of text classification systems. LSAP incorporates label semantics into pre-trained generative models (T5 in our case) by performing secondary pre-training on labeled sentences from a variety of domains. As domain-general pre-training requires large amounts of data, we develop a filtering and labeling pipeline to automatically create sentence-label pairs from unlabeled text. We perform experiments on intent (ATIS, Snips, TOPv2) and topic classification (AG News, Yahoo! Answers). LSAP obtains significant accuracy improvements over state-of-the-art models for few-shot text classification while maintaining performance comparable to state of the art in high-resource settings.
Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Making

Many important decisions in daily life are made with the help of advisors, e.g., decisions about medical treatments or financial investments. Whereas in the past, advice has often been received from human experts, friends, or family, advisors based on artificial intelligence (AI) have become more and more present nowadays. Typically, the advice generated by AI is judged by a human and either deemed reliable or rejected. However, recent work has shown that AI advice is not always beneficial, as humans have shown to be unable to ignore incorrect AI advice, essentially representing an over-reliance on AI. Therefore, the aspired goal should be to enable humans not to rely on AI advice blindly but rather to distinguish its quality and act upon it to make better decisions. Specifically, that means that humans should rely on the AI in the presence of correct advice and self-rely when confronted with incorrect advice, i.e., establish appropriate reliance (AR) on AI advice on a case-by-case basis. Current research lacks a metric for AR. This prevents a rigorous evaluation of factors impacting AR and hinders further development of human-AI decision-making. Therefore, based on the literature, we derive a measurement concept of AR. We propose to view AR as a two-dimensional construct that measures the ability to discriminate advice quality and behave accordingly. In this article, we derive the measurement concept, illustrate its application and outline potential future research.
Polish spaces of Banach spaces. Complexity of isometry and isomorphism classes

We study the complexities of isometry and isomorphism classes of separable Banach spaces in the Polish spaces of Banach spaces recently introduced and investigated by the authors in [14]. We obtain sharp results concerning the most classical separable Banach spaces. We prove that the inﬁnite-dimensional separable Hilbert space is character- ized as the unique separable inﬁnite-dimensional Banach space whose isometry class is closed, and also as the unique separable inﬁnite-dimensional Banach space whose isomorphism class is F σ . For p ∈ [1 , 2) ∪ (2 , ∞ ), we show that the isometry classes of L p [0 , 1] and ℓ p are G δ -complete sets and F σδ -complete sets, respectively. Then we show that the isometry class of c 0 is an F σδ -complete set. Additionally, we compute the complexities of many other natural classes of separable Banach spaces; for instance, the class of separable L p,λ + -spaces, for p,λ ≥ 1, is shown to be a G δ -set, the class of superreﬂexive spaces is shown to be an F σδ -set, and the class of spaces with local Π-basis structure is shown to be a Σ 06 -set. The paper is concluded with many open problems and suggestions for a future research.
GPT-NeoX-20B: An Open-Source Autoregressive Language Model

We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. In this work, we describe GPT-NeoX-20B’s architecture and training, and evaluate its performance. We open-source the training and evaluation code, as well as the model weights, at https://github.com/EleutherAI/gpt-neox.
Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin-picking

In this paper, we propose an iterative self-training framework for sim-to-real 6D object pose estimation to facilitate cost-effective robotic grasping. Given a bin-picking scenario, we establish a photo-realistic simulator to synthesize abundant virtual data, and use this to train an initial pose estimation network. This network then takes the role of a teacher model, which generates pose predictions for unlabeled real data. With these predictions, we further design a comprehensive adaptive selection scheme to distinguish reliable results, and leverage them as pseudo labels to update a student model for pose estimation on real data. To continuously improve the quality of pseudo labels, we iterate the above steps by taking the trained student model as a new teacher and re-label real data using the refined teacher model. We evaluate our method on a public benchmark and our newly-released dataset, achieving an ADD(-S) improvement of 11.49% and 22.62% respectively. Our method is also able to improve robotic bin-picking success by 19.54%, demonstrating the potential of iterative sim-to-real solutions for robotic applications.