Miracle-dz/UniFAR-FTUs huggingface dataset:
Do language models have coherent mental models of everyday things?

When people think of everyday things like an egg, they typically have a mental image associated with it. This allows them to correctly judge, for example, that “the yolk surrounds the shell” is a false statement. Do language models similarly have a coherent picture of such everyday things? To investigate this, we propose a benchmark dataset consisting of 100 everyday things, their parts, and the relationships between these parts, expressed as 11,720 “X relation Y? ” true/false questions. Using these questions as probes, we observe that state-of-the-art pre-trained language models (LMs) like GPT-3 and Macaw have fragments of knowledge about these everyday things, but do not have fully coherent “parts mental models” (54-59% accurate, 19-43% conditional constraint violation). We propose an extension where we add a constraint satisfaction layer on top of the LM’s raw predictions to apply commonsense constraints. As well as removing inconsistencies, we find that this also significantly improves accuracy (by 16-20%), suggesting how the incoherence of the LM’s pictures of everyday things can be significantly reduced.
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters

Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning abilities of large language models (LLMs). CoT explicitly encourages the LLM to generate intermediate rationales for solving a problem, by providing a series of reasoning steps in the demonstrations. Despite its success, there is still little understanding of what makes CoT prompting effective and which aspects of the demonstrated reasoning steps contribute to its performance. In this paper, we show that CoT reasoning is possible even with invalid demonstrations - prompting with invalid reasoning steps can achieve over 80-90% of the performance obtained using CoT under various metrics, while still generating coherent lines of reasoning during inference. Further experiments show that other aspects of the rationales, such as being relevant to the query and correctly ordering the reasoning steps, are much more important for effective CoT reasoning. Overall, these findings both deepen our understanding of CoT prompting, and open up new questions regarding LLMs’ capability to learn to reason in context.
Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languages

We present, Naamapadam, the largest publicly available Named Entity Recognition (NER) dataset for the 11 major Indian languages from two language families. The dataset contains more than 400k sentences annotated with a total of at least 100k entities from three standard entity categories (Person, Location, and, Organization) for 9 out of the 11 languages. The training dataset has been automatically created from the Samanantar parallel corpus by projecting automatically tagged entities from an English sentence to the corresponding Indian language translation. We also create manually annotated testsets for 9 languages. We demonstrate the utility of the obtained dataset on the Naamapadam-test dataset. We also release IndicNER, a multilingual IndicBERT model fine-tuned on Naamapadam training set. IndicNER achieves an F1 score of more than 80 for 7 out of 9 test languages. The dataset and models are available under open-source licences at https://ai4bharat.iitm. ac.in/naamapadam.
General Prediction Relating Yes‐No and Forced‐Choice Results

One reason for developing detailed models of the detection or discrimination process is to provide a means of predicting performance in a variety of test situations, thereby permitting a psychoacoustic theory that is not bound to any single psychophysical method. Two general methods are often used to study discrimination or detection processes. In the Yes‐No task, a single observation interval occurs; the signal is either presented or not and the subject says that Yes—he hears the signal or No—he does not. In the forced‐choice task, several observation intervals occur; the signal is presented in only one of them and the subject indicates the interval in which he believes the signal has occurred. It can be demonstrated that a number of psychophysical models (e.g., threshold, detection theory) all agree in predicting the same relation between the accuracy of the judgments in the forced‐choice task and the area under the Yes‐No ROC curve. (The ROC curve is a plot of false‐alarm rate versus hit rate for diffe.
Do I have the Knowledge to Answer? Investigating Answerability of Knowledge Base Questions

When answering natural language questions over knowledge bases, missing facts, incomplete schema and limited scope naturally lead to many questions being unanswerable. While answerability has been explored in other QA settings, it has not been studied for QA over knowledge bases (KBQA). We create GrailQAbility, a new benchmark KBQA dataset with unanswerability, by first identifying various forms of KB incompleteness that make questions unanswerable, and then systematically adapting GrailQA (a popular KBQA dataset with only answerable questions). Experimenting with three state-of-the-art KBQA models, we find that all three models suffer a drop in performance even after suitable adaptation for unanswerable questions. In addition, these often detect unanswerability for wrong reasons and find specific forms of unanswerability particularly difficult to handle. This underscores the need for further research in making KBQA systems robust to unanswerability.
Influence of boundary-layer disturbances on the instability of a roughness wake in a high-speed boundary layer

Abstract The excitation of instability modes in the wake generated behind a discrete roughness element in a boundary layer at Mach 6 is analysed through numerical simulations of the compressible Navier–Stokes equations. Recent experimental observations show that transition to turbulence in high-speed boundary layers during re-entry flight is dominated by wall roughness effects. Therefore, understanding the roughness-induced transition to turbulence in this flow regime is of primary importance. Our results show that a discrete roughness element with a height of about half the local boundary-layer thickness generates an unstable wake able to sustain the growth of a number of modes. The most unstable of these modes are a sinuous mode (mode SL) and two varicose modes (modes VL and VC). The varicose modes grow approximately 17 % faster than the most unstable Mack mode and their growth persists over a longer streamwise distance, thereby leading to a notable acceleration of the laminar–turbulent transition process. Two main mechanisms are identified for the excitation of wake modes: the first is based on the interaction between the external disturbances and the reverse flow regions induced by the roughness element and the second is due to the interaction between the boundary-layer modes (first modes and Mack modes) and the non-parallel roughness wake. An important finding of the present study is that, while being less unstable, mode SL is the preferred instability for the first of the above excitation mechanisms, which drives the wake modes excitation in the absence of boundary-layer modes. Modes VL and VC are excited through the second mechanism and, hence, become important when first modes and Mack modes come into interaction with the roughness wake. The new mode VC presents similarities with the Mack mode instability, including the tuning between its most unstable wavelength and the local boundary-layer thickness, and it is believed to play a fundamental role in the roughness-induced transition of high-speed boundary layers. In contrast to the smooth-wall case, wall cooling is stabilising for all the roughness-wake modes.
A Comprehensive Study and Comparison of the Robustness of 3D Object Detectors Against Adversarial Attacks

Deeplearning-based3Dobjectdetectorshavemade signiﬁcant progress in recent years and have been deployed in a wide range of applications. It is crucial to understand the robustness of detectors against adversarial attacks when employing detectors in security-critical applications. In this paper, we make the ﬁrst attempt to conduct a thorough evaluation and analysis of the robustness of 3D detectors under adversarial attacks. Speciﬁcally, we ﬁrst extend three kinds of adversarial attacks to the 3D object detection task to benchmark the robustness of state-of-the-art 3D object detectors against attacks on KITTI and Waymo datasets, subsequently followed by the analysis of the relationship between robustness and properties of detectors. Then, we explore the trans-ferability of cross-model, cross-task, and cross-data attacks. We ﬁnally conduct comprehensive experiments of defense for 3D detectors, demonstrating that simple transformations like ﬂipping are of little help in improving robustness when the strategy of transformation imposed on input point cloud data is exposed to attackers. Our ﬁndings will facilitate investigations in understanding and defending the adversarial attacks against 3D object detectors to advance this ﬁeld.
Texture Representation via Analysis and Synthesis with Generative Adversarial Networks

We investigate data-driven texture modeling via analysis and synthesis with generative adversarial networks. For network training and testing, we have compiled a diverse set of spatially homogeneous textures, ranging from stochastic to regular. We adopt StyleGAN3 for synthesis and demonstrate that it produces diverse textures beyond those represented in the training data. For texture analysis, we propose GAN inversion using a novel latent domain reconstruction consistency criterion for synthesized textures, and iterative refinement with Gramian loss for real textures. We propose perceptual procedures for evaluating network capabilities, exploring the global and local behavior of latent space trajectories, and comparing with existing texture analysis-synthesis techniques.
Analytical and numerical development for the two-centre overlap-like quantum similarity integrals over Slater-type functions

In molecular quantum similarity measure between two molecules, using the molecular electron density, the major task involves the accurate numerical evaluation of overlap between the two electron densities in the integral measure. By expanding the electron densities in terms of atomic orbitals, using the linear combination of atomic orbitals approach, a large number of overlap-like quantum similarity integrals over the basis functions in the two molecules will be required accurately for the calculation of a meaningful quantum similarity measure. Improvement of the computational methods of these integrals would be indispensable to a further development in computational studies of large molecular systems. Analytic expressions were obtained for these overlap-like quantum similarity integrals over Slater-type functions, in terms of the usual two-centre overlap integrals over B functions. Different approaches were developed for the numerical evaluation of these two-centre overlap integrals over B functions, which can be used for the numerical evaluation of the two-centre overlap-like quantum similarity integrals over Slater-type functions. In this work we present our approach which is based on the use of nonlinear transformations for improving convergence of highly oscillatory integrals. In the case of more complicated multicentre integrals, this approach is shown to be able to produce remarkably good results. We also present different representations, which were obtained by Steinborn group, and which can be used for a fast and accurate computation of the two-centre overlap-like quantum similarity integrals over Slater-type functions.
Twisted-bilayer FeSe and the Fe-based superlattices

We derive BM-like continuum models for the bands of superlattice heterostructures formed out of Fe-chalcogenide monolayers: (I) a single monolayer experiencing an external periodic potential, and (II) twisted bilayers with long-range moire tunneling. A symmetry derivation for the inter-layer moire tunnelling is provided for both the \GammaΓ and MM high-symmetry points. In this paper, we focus on moire bands formed from hole-band maxima centered on \GammaΓ, and show the possibility of moire bands with C=0C=0 or ±1±1 topological quantum numbers without breaking time-reversal symmetry. In the C=0C=0 region for \theta→0θ→0 (and similarly in the limit of large superlattice period for I), the system becomes a square lattice of 2D harmonic oscillators. We fit our model to FeSe and argue that it is a viable platform for the simulation of the square Hubbard model with tunable interaction strength.
Tokenization Consistency Matters for Generative Models on Extractive NLP Tasks

Generative models have been widely applied to solve extractive tasks, where parts of the input is extracted to form the desired output, and achieved significant success. For example, in extractive question answering (QA), generative models have constantly yielded state-of-the-art results. In this work, we identify the issue of tokenization inconsistency that is commonly neglected in training these models. This issue damages the extractive nature of these tasks after the input and output are tokenized inconsistently by the tokenizer, and thus leads to performance drop as well as hallucination. We propose a simple yet effective fix to this issue and conduct a case study on extractive QA. We show that, with consistent tokenization, the model performs better in both in-domain and out-of-domain datasets, with a notable average of +1.7 F2 gain when a BART model is trained on SQuAD and evaluated on 8 QA datasets. Further, the model converges faster, and becomes less likely to generate out-of-context answers. With these findings, we would like to call for more attention on how tokenization should be done when solving extractive tasks and recommend applying consistent tokenization during training.
Detecting Contradictory COVID-19 Drug Efficacy Claims from Biomedical Literature

The COVID-19 pandemic created a deluge of questionable and contradictory scientific claims about drug efficacy – an “infodemic” with lasting consequences for science and society. In this work, we argue that NLP models can help domain experts distill and understand the literature in this complex, high-stakes area. Our task is to automatically identify contradictory claims about COVID-19 drug efficacy. We frame this as a natural language inference problem and offer a new NLI dataset created by domain experts. The NLI framing allows us to create curricula combining existing datasets and our own. The resulting models are useful investigative tools. We provide a case study of how these models help a domain expert summarize and assess evidence concerning remdisivir and hydroxychloroquine.
EIT: Enhanced Interactive Transformer

In this paper, we propose a novel architecture, the Enhanced Interactive Transformer (EIT), to address the issue of head degradation in self-attention mechanisms. Our approach replaces the traditional multi-head self-attention mechanism with the Enhanced Multi-Head Attention (EMHA) mechanism, which relaxes the one-to-one mapping constraint among queries and keys, allowing each query to attend to multiple keys. Furthermore, we introduce two interaction models, Inner-Subspace Interaction and Cross-Subspace Interaction, to fully utilize the many-to-many mapping capabilities of EMHA. Extensive experiments on a wide range of tasks (e.g. machine translation, abstractive summarization, grammar correction, language modelling and brain disease automatic diagnosis) show its superiority with a very modest increase in model size.
Bell inequalities in 2-2 scattering

We consider Bell inequalities in 2-2 scattering of photons, gravitons, fermions and pions. We choose measurement settings that give maximum Bell violation for maximally entangled states and calculate the relevant Bell inequalities for these processes. For photon scattering at low energies, QED exhibits Bell violation for all scattering angles except for a small transverse region. This leads to a fine-tuning problem. Incorporating a light axion/axion-like particle (ALP) removes the fine-tuning problem and constrains the axion-coupling--axion-mass parameters. Allowing for graviton exchange and demanding Bell violation in photon scattering, we find that the Weak Gravity Conjecture is satisfied. Quantum gravity effect on axion coupling is discussed. For 2-2 graviton scattering, we find that CEMZ bounds allow for at most small Bell violations. Restriction on the Weinberg angle is found by demanding Bell violation in Bhabha scattering. We use recent S-matrix bootstrap data for pions and photons to study the Bell parameter in the space of allowed S-matrices. In the photon case, we study the Bell parameters as a function of energy and find support for the EFT observations. We discuss Bell parameter for pion S-matrices, which are qutrits. For pions, we find that there is a minimization of a suitable Bell parameter for S-matrices which exhibit Regge behaviour.
Document-level Relation Extraction with Relation Correlations

Document-level relation extraction faces two overlooked challenges: long-tail problem and multi-label problem. Previous work focuses mainly on obtaining better contextual representations for entity pairs, hardly address the above challenges. In this paper, we analyze the co-occurrence correlation of relations, and introduce it into DocRE task for the first time. We argue that the correlations can not only transfer knowledge between data-rich relations and data-scarce ones to assist in the training of tailed relations, but also reflect semantic distance guiding the classifier to identify semantically close relations for multi-label entity pairs. Specifically, we use relation embedding as a medium, and propose two co-occurrence prediction sub-tasks from both coarse- and fine-grained perspectives to capture relation correlations. Finally, the learned correlation-aware embeddings are used to guide the extraction of relational facts. Substantial experiments on two popular DocRE datasets are conducted, and our method achieves superior results compared to baselines. Insightful analysis also demonstrates the potential of relation correlations to address the above challenges.
Probabilistic Quantile Factor Analysis

This paper extends quantile factor analysis to a probabilistic variant that incorporates regularization and computationally efficient variational approximations. By means of synthetic and real data experiments it is established that the proposed estimator can achieve, in many cases, better accuracy than a recently proposed loss-based estimator. We contribute to the literature on measuring uncertainty by extracting new indexes of low, medium and high economic policy uncertainty, using the probabilistic quantile factor methodology. Medium and high indexes have clear contractionary effects, while the low index is benign for the economy, showing that not all manifestations of uncertainty are the same.
Does CLIP Bind Concepts? Probing Compositionality in Large Image Models

Large-scale neural network models combining text and images have made incredible progress in recent years. However, it remains an open question to what extent such models encode compositional representations of the concepts over which they operate, such as correctly identifying ‘red cube’ by reasoning over the constituents ‘red’ and ‘cube’. In this work, we focus on the ability of a large pretrained vision and language model (CLIP) to encode compositional concepts and to bind variables in a structure-sensitive way (e.g., differentiating ‘cube behind sphere’ from ‘sphere behind cube’). To inspect the performance of CLIP, we compare several architectures from research on compositional distributional semantics models (CDSMs), a line of research that attempts to implement traditional compositional linguistic structures within embedding spaces. We benchmark them on three synthetic datasets – single-object, two-object, and relational – designed to test concept binding. We find that CLIP can compose concepts in a single-object setting, but in situations where concept binding is needed, performance drops dramatically. At the same time, CDSMs also perform poorly, with best performance at chance level.
LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer

Graphic layout designs play an essential role in visual communication. Yet handcrafting layout designs is skill-demanding, time-consuming, and non-scalable to batch production. Generative models emerge to make design automation scalable but it remains non-trivial to produce designs that comply with designers' multimodal desires, i.e., constrained by background images and driven by foreground content. We propose LayoutDETR that inherits the high quality and realism from generative modeling, while reformulating content-aware requirements as a detection problem: we learn to detect in a background image the reasonable locations, scales, and spatial relations for multimodal foreground elements in a layout. Our solution sets a new state-of-the-art performance for layout generation on public benchmarks and on our newly-curated ad banner dataset. We integrate our solution into a graphical system that facilitates user studies, and show that users prefer our designs over baselines by significant margins. Our code, models, dataset, graphical system, and demos are available at https://github.com/salesforce/LayoutDETR.
Towards Reasoning in Large Language Models: A Survey

Reasoning is a fundamental aspect of human intelligence that plays a crucial role in activities such as problem solving, decision making, and critical thinking. In recent years, large language models (LLMs) have made significant progress in natural language processing, and there is observation that these models may exhibit reasoning abilities when they are sufficiently large. However, it is not yet clear to what extent LLMs are capable of reasoning. This paper provides a comprehensive overview of the current state of knowledge on reasoning in LLMs, including techniques for improving and eliciting reasoning in these models, methods and benchmarks for evaluating reasoning abilities, findings and implications of previous research in this field, and suggestions on future directions. Our aim is to provide a detailed and up-to-date review of this topic and stimulate meaningful discussion and future work.
Distribution of Ruelle resonances for real-analytic Anosov diffeomorphisms

We prove an upper bound for the number of Ruelle resonances for Koopman operators associated to real-analytic Anosov diffeomorphisms: in dimension d, the number of resonances larger than r is a O(| log r|) when r goes to 0. For each connected component of the space of realanalytic Anosov diffeomorphisms on a real-analytic manifold, we prove a dichotomy: either the exponent d in our bound is never optimal, or it is optimal on a dense subset. Using examples constructed by Bandtlow, Just and Slipantschuk, we see that we are always in the latter situation for connected components of the space of real-analytic Anosov diffeomorphisms on the 2-dimensional torus.
The stability of nonequilibrium molecular dynamics simulations of elongational flows

We show that nonequilibrium molecular dynamics simulations of elongational flows are inherently unstable over long periods of time. This instability leads to a catastrophic nonequilibrium phase transition that destroy the true structure of the fluid. We identify the source of this instability as a lack of momentum conservation, resulting from numerical round-off errors. We show that this error grows exponentially in the direction of compression, and present two numerical recipes that involve only minor perturbations to the particle trajectories to guarantee momentum conservation.