I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.

Title: Gauge fixing, BRS invariance and Ward identities for randomly stirred flows

Abstract: The Galilean invariance of the Navier-Stokes equation is shown to be akin to a global gauge symmetry familiar from quantum field theory. This symmetry leads to a multiple counting of infinitely many inertial reference frames in the path integral approach to randomly stirred fluids. This problem is solved by fixing the gauge, i.e., singling out one reference frame. The gauge fixed theory has an underlying Becchi-Rouet-Stora (BRS) symmetry which leads to the Ward identity relating the exact inverse response and vertex functions. This identification of Galilean invariance as a gauge symmetry is explored in detail, for different gauge choices and by performing a rigorous examination of a discretized version of the theory. The Navier-Stokes equation is also invariant under arbitrary rectilinear frame accelerations, known as extended Galilean invariance (EGI). We gauge fix this extended symmetry and derive the generalized Ward identity that follows from the BRS invariance of the gauge-fixed theory. This new Ward identity reduces to the standard one in the limit of zero acceleration. This gauge-fixing approach unambiguously shows that Galilean invariance and EGI constrain only the zero mode of the vertex but none of the higher wavenumber modes.


Please find me the paper on training code large language models to follow instructions via git commits.

Title: Search for gravitational waves from binary black hole inspirals in LIGO data

Abstract: We report on a search for gravitational waves from binary black hole inspirals in the data from the second science run of the LIGO interferometers. The search focused on binary systems with component masses between 3 and 20 solar masses. Optimally oriented binaries with distances up to 1 Mpc could be detected with efficiency of at least 90%. We found no events that could be identified as gravitational waves in the 385.6 hours of data that we searched.

Title: Is the Commercial Mass Media Necessary, or Even Desirable, for Liberal Democracy?

Abstract: Is a commercial mass media, dependent on the market for its sustenance, necessary, or even desirable, for liberal democracy? Yochai Benkler has argued that a decentralized, peer-to-peer system of communications and information is both possible with digital technology and preferable to a system based on commercial mass media. He has contended in fact that the presence of politically powerful, copyright-rich mass media imposes significant barriers to the development of peer-to-peer information-sharing networks. In contrast, I have argued that the commercial mass media play an important, and perhaps even vital, role in liberal democracy by galvanizing public opinion, serving as a watchdog against government and corporate wrongdoing, agenda-setting (which enables public discourse), and serving as a relatively trustworthy source of information. This paper seeks to push the ball forward on this issue. It first isolates and enumerates the contributions that the commercial mass media are said to make towards liberal democracy. It then briefly assesses the extent to which the commercial mass media actually fulfills these constitutive functions. It then asks whether alternative institutions might serve some or all of these functions just as well or better. In so doing, it looks both to the past and the future. First, it examines the political party-supported press that thrived in the United States through much of the 19th century. Second, it examines government-funded mass media. Third, it looks, skeptically, at possibilities for peer-to-peer sharing of information and opinion in the digital network environment. I conclude that, despite the weaknesses of commercial mass media, an information policy suitable to liberal democracy should include a plurality of types of voices, including commercial mass media.


Title: InstructCoder: Instruction Tuning Large Language Models for Code Editing

Abstract: Code editing encompasses a variety of pragmatic tasks that developers deal with daily. Despite its relevance and practical usefulness, automatic code editing remains an underexplored area in the evolution of deep learning models, partly due to data scarcity. In this work, we explore the use of Large Language Models (LLMs) to edit code based on user instructions. Evaluated on a novel human-written execution-based benchmark dubbed EditEval, we found current models often struggle to fulfill the instructions. In light of this, we contribute InstructCoder, the first instruction-tuning dataset designed to adapt LLMs for general-purpose code editing, containing high-diversity code-editing tasks such as comment insertion, code optimization, and code refactoring. It consists of over 114,000 instruction-input-output triplets and covers multiple distinct code editing scenarios. The collection process starts with filtered commit data sourced from GitHub Python repositories as seeds. Subsequently, the dataset is systematically expanded through an iterative process, where both seed and generated tasks are used to prompt ChatGPT for more data. Our findings reveal that open-source LLMs fine-tuned on InstructCoder can significantly enhance the accuracy of code edits, exhibiting superior code-editing performance matching advanced proprietary LLMs. The datasets and the source code are publicly available at https://github.com/qishenghu/CodeInstruct.


do you need a visa for a connecting flight in vietnam?

Meta Stackexchange

Q: Stack Exchange site to ask Visa and passport related questions? Where can I ask questions about Passports and Visas?
Is there a Stack Exchange site for this?

A: travel's help centre says you can ask about visas.

customs & border crossing, visas and other travel documents such as permits

They also have a passports tag.
The Expatriates Stack Exchange is also available for people living or intend to live abroad on a long-term basis.
Do check out the help centre text of these sites before you ask to make sure your question is on topic.

Stackoverflow Stackexchange

Q: How to load data from splitted gzip files into redshift? Can I load data from splitted gzip files into an amazon redshift table?
I can load data from single gzip file or from splitted text files.
But can I load data from splitted gzip files?

A: I'm assuming here that you mean that you have multiple CSV files that are each gzipped.
First, upload each file to an S3 bucket under the same prefix and delimiter.
s3://S3_BUCKET/S3_PREFIX/file0.gz
s3://S3_BUCKET/S3_PREFIX/file1.gz
Then, execute the Redshift copy command:
copy TABLE_NAME from 's3://S3_BUCKET/S3_PREFIX' credentials 'aws_access_key_id=ACCESS_KEY;aws_secret_access_key=SECRET_KEY' csv gzip

*

*specify the S3 bucket and prefix

*include credentials that have permission to read the S3 objects

*ensure that the destination table already exists and has columns compatible with the CSV

*specify the csv and gzip options

Also, it's a good idea to have your file count proportional to the number of nodes in your cluster.


How to reduce overoptimization in reinforcement learning from human feedback?

Title: Reward Model Ensembles Help Mitigate Overoptimization

Abstract: Reinforcement learning from human feedback (RLHF) is a standard approach for fine-tuning large language models to follow instructions. As part of this process, learned reward models are used to approximately model human preferences. However, as imperfect representations of the "true" reward, these learned reward models are susceptible to overoptimization. Gao et al. (2023) studied this phenomenon in a synthetic human feedback setup with a significantly larger "gold" reward model acting as the true reward (instead of humans) and showed that overoptimization remains a persistent problem regardless of the size of the proxy reward model and training data used. Using a similar setup, we conduct a systematic study to evaluate the efficacy of using ensemble-based conservative optimization objectives, specifically worst-case optimization (WCO) and uncertainty-weighted optimization (UWO), for mitigating reward model overoptimization when using two optimization methods: (a) best-of-n sampling (BoN) (b) proximal policy optimization (PPO). We additionally extend the setup of Gao et al. (2023) to include 25% label noise to better mirror real-world conditions. Both with and without label noise, we find that conservative optimization practically eliminates overoptimization and improves performance by up to 70% for BoN sampling. For PPO, ensemble-based conservative optimization always reduces overoptimization and outperforms single reward model optimization. Moreover, combining it with a small KL penalty successfully prevents overoptimization at no performance cost. Overall, our results demonstrate that ensemble-based conservative optimization can effectively counter overoptimization.

Title: Scalable Ensembling For Mitigating Reward Overoptimisation

Abstract: Reinforcement Learning from Human Feedback (RLHF) has enabled significant advancements within language modeling for powerful, instruction-following models. However, the alignment of these models remains a pressing challenge as the policy tends to overfit the learned ``proxy" reward model past an inflection point of utility as measured by a ``gold" reward model that is more performant -- a phenomenon known as overoptimisation. Prior work has mitigated this issue by computing a pessimistic statistic over an ensemble of reward models, which is common in Offline Reinforcement Learning but incredibly costly for language models with high memory requirements, making such approaches infeasible for sufficiently large models. To this end, we propose using a shared encoder but separate linear heads. We find this leads to similar performance as the full ensemble while allowing tremendous savings in memory and time required for training for models of similar size.


what is the difference between domain users and authenticated users?

Stackoverflow Stackexchange

Q: Erlang: robustness against port owner's death What happens when the process which owns a spawned port dies and is restarted by the supervisor?

*

*Is there a way for the old port not to die with its owner and for the new owner to "take over"?

*Failing that, is it possible to ensure that the spawned process terminates when its port dies?

A: First, notice that you don't want the port owner to die. So move any "dangerous" code out to another process and make the port owner as dumb as possible. This is error-mitigation the Erlang way. Now, if that process dies, something is really bad, so in that case it may be clever to recycle the port as well. But since we moved everything out, we are counting on that not happening.
Regarding 2, the port will send a specific message when it terminates, so you can arrange for your spawned process to gracefully detect this and terminate with it. See
http://www.erlang.org/doc/reference_manual/ports.html

Stackoverflow Stackexchange

Q: What are the differences between LDAP and Active Directory authentication? I have tried to search without success.
(I do know difference between LDAP and Active Directory. Also read this What are the differences between LDAP and Active Directory?)
According to my understanding, LDAP authenticates (binds) with DN (distinguish name) and password.
E.g. CN=bob, OU=Users,DC=test,DC=com
Active Directory authenticates with a short user name or full domain user name and password.
E.g. bob or bob@test.com
Therefore, LDAP does not allow to users with the same CN in the same tree (OU) but it allows the same CN in a different OU.
Active Directory does not allow the same user name in the same domain, even the user object located in different tree of Active Directory.
Active Directory allows the same user name in different domains.
Other differences?

A: Active Directory is a product and a directory service. LDAP is an application layer protocol and an IETF standard. Active Directory supports and implements LDAP, therefore allowing RFC-compliant LDAP clients to talk to it and perform LDAP operations such as bind, search, etc.


Title: Gauge fixing, BRS invariance and Ward identities for randomly stirred flows

Abstract: The Galilean invariance of the Navier-Stokes equation is shown to be akin to a global gauge symmetry familiar from quantum field theory. This symmetry leads to a multiple counting of infinitely many inertial reference frames in the path integral approach to randomly stirred fluids. This problem is solved by fixing the gauge, i.e., singling out one reference frame. The gauge fixed theory has an underlying Becchi-Rouet-Stora (BRS) symmetry which leads to the Ward identity relating the exact inverse response and vertex functions. This identification of Galilean invariance as a gauge symmetry is explored in detail, for different gauge choices and by performing a rigorous examination of a discretized version of the theory. The Navier-Stokes equation is also invariant under arbitrary rectilinear frame accelerations, known as extended Galilean invariance (EGI). We gauge fix this extended symmetry and derive the generalized Ward identity that follows from the BRS invariance of the gauge-fixed theory. This new Ward identity reduces to the standard one in the limit of zero acceleration. This gauge-fixing approach unambiguously shows that Galilean invariance and EGI constrain only the zero mode of the vertex but none of the higher wavenumber modes.

Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Title: Text and Code Embeddings by Contrastive Pre-Training

Abstract: Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture. In this work, we show that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code. The same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine-tuned models. On linear-probe classification accuracy averaging over 7 tasks, our best unsupervised model achieves a relative improvement of 4% and 1.8% over previous best unsupervised and supervised text embedding models respectively. The same text embeddings when evaluated on large-scale semantic search attains a relative improvement of 23.4%, 14.7%, and 10.6% over previous best unsupervised methods on MSMARCO, Natural Questions and TriviaQA benchmarks, respectively. Similarly to text embeddings, we train code embedding models on (text, code) pairs, obtaining a 20.8% relative improvement over prior best work on code search.


Meta Stackexchange

Q: Google time and answer mistakes Take a look a this :

On the google search page I can see

*

*13 hours ago

*1 answer

When I actually posted the question :

*

*8 mins ago

*0 answers

This should be fixed.

Gaming Stackexchange

Q: Do Nirnroots regrow in Skyrim? In Oblivion, Nirnroots were a collectible - there was a finite number of them*, and they did not respawn.
I've been noticing, however, that when I pick Nirnroots in Skyrim, the roots of the plant remain behind. This got me wondering...
Do Nirnroots regrow in Skyrim?
What about the Crimson Nirnroots of Blackreach?
*discounting the respawning food-nirnroot in Frostcrag Spire

A: I can confirm they grow back after 15 days also. I'm using the PS3 version and picked about 20 of them before doing the related quest. It does give you a negative number if you try dropping them and picking them back up. So, I was forced to wait 15 days to pick a total of 50 of them.


Scifi Stackexchange

Q: Why did Star Trek: The Next Generation season 2 only have 22 episodes? In the Star Trek franchise shows Star Trek: The Next Generation, Star Trek: Deep Space Nine and Star Trek: Voyager, each show almost always had 26 episodes per season.  Besides Voyager's first season, the only exception in this is the second season of Star Trek: The Next Generation, which instead had 22.  Why is this?

A: There was a writers' strike in 1988.
The Writers´ Guild of America went out on strike in the spring and summer of 1988. They demanded more creative control and residuals for the shows they wrote.
The writers of ST:TNG at Paramount joined in despite their Sacred Duty.
The strike ended on August 7, but the production pipeline was adversely affected, and ST:TNG wasn't able to start airing until November 21. (This episode, "The Child", contained the first appearance of Whoopi Goldberg playing Guinan.)

Mathematica Stackexchange

Q: how to download Finite ELements tutorial as PDF For long tutorials, I prefer hard copy.
Is it possible to download from somewhere this document
https://reference.wolfram.com/language/FEMDocumentation/tutorial/FiniteElementOverview.html
as PDF file?  Few years ago WRI used to have PDF files for all its documentations, but I am not able to find one for the above now.


Title: SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages

Abstract: Southeast Asia (SEA) is a region rich in linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA, compromising the quality of AI models for SEA languages. Evaluating models for SEA languages is challenging due to the scarcity of high-quality datasets, compounded by the dominance of English training data, raising concerns about potential cultural misrepresentation. To address these challenges, we introduce SEACrowd, a collaborative initiative that consolidates a comprehensive resource hub that fills the resource gap by providing standardized corpora in nearly 1,000 SEA languages across three modalities. Through our SEACrowd benchmarks, we assess the quality of AI models on 36 indigenous languages across 13 tasks, offering valuable insights into the current AI landscape in SEA. Furthermore, we propose strategies to facilitate greater AI advancements, maximizing potential utility and resource equity for the future of AI in SEA.

Title: Carriers of \emph{Sargassum} and mechanism for coastal inundation in the Caribbean Sea

Abstract: We identify effective carriers of \emph{Sargassum} in the Caribbean Sea and describe a mechanism for coastal choking. Revealed from satellite altimetry, the carriers of \emph{Sargassum} are mesoscale eddies (vortices of 50-km radius or larger) with coherent material (i.e., fluid) boundaries. These are observer-independent -- \emph{unlike} eddy boundaries identified with instantaneously closed streamlines of the altimetric sea-surface height field -- and furthermore harbor finite-time attractors for networks of elastically connected finite-size buoyant or "inertial" particles dragged by ocean currents and winds, a mathematical abstraction of \emph{Sargassum} rafts. The mechanism of coastal inundation, identified using a minimal model of surface-intensified Caribbean Sea eddies, is thermal instability in the presence of bottom topography.


Paper that shows multitask finetuned language models can generalize to unseen languages

Title: Constructive Complexity and Artificial Reality: An Introduction

Abstract: Basic problems of complex systems are outlined with an emphasis on irreducibility and dynamic many-to-many correspondences. We discuss the importance of a constructive approach to artificial reality and the significance of an internal observer.

Title: The Absence of Positive Energy Bound States for a Class of Nonlocal Potentials

Abstract: We generalize in this paper a theorem of Titchmarsh for the positivity of Fourier sine integrals. We apply then the theorem to derive simple conditions for the absence of positive energy bound states (bound states embedded in the continuum) for the radial Schr\"odinger equation with nonlocal potentials which are superposition of a local potential and separable potentials.


Title: a survey on GPT-3

Abstract: This paper provides an introductory survey to GPT-3. We cover some of the historical development behind this technology, some of the key features of GPT-3, and discuss the machine learning model and the datasets used. We survey both academic and commercial efforts applying GPT-3 in diverse domains such as developing conversational AI chatbots, software development, creative work, domain knowledge, and business productivity. We discuss some of the challenges that GPT-3 faces such as the problems of training complexity, bias, and hallucination/incorrect answers. We also discuss the future research opportunities in this area.

Title: Improving Short Text Classification With Augmented Data Using GPT-3

Abstract: GPT-3 is a large-scale natural language model developed by OpenAI that can perform many different tasks, including topic classification. Although researchers claim that it requires only a small number of in-context examples to learn a task, in practice GPT-3 requires these training examples to be either of exceptional quality or a higher quantity than easily created by hand. To address this issue, this study teaches GPT-3 to classify whether a question is related to data science by augmenting a small training set with additional examples generated by GPT-3 itself. This study compares two classifiers: the GPT-3 Classification Endpoint with augmented examples, and the GPT-3 Completion Endpoint with an optimal training set chosen using a genetic algorithm. We find that while the augmented Completion Endpoint achieves upwards of 80 percent validation accuracy, using the augmented Classification Endpoint yields more consistent accuracy on unseen examples. In this way, giving large-scale machine learning models like GPT-3 the ability to propose their own additional training examples can result in improved classification performance.


paper showing crosslingual generalization is possible

Title: Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization

Abstract: Large language models (LLMs) have exhibited considerable cross-lingual generalization abilities, whereby they implicitly transfer knowledge across languages. However, the transfer is not equally successful for all languages, especially for low-resource ones, which poses an ongoing challenge. It is unclear whether we have reached the limits of implicit cross-lingual generalization and if explicit knowledge transfer is viable. In this paper, we investigate the potential for explicitly aligning conceptual correspondence between languages to enhance cross-lingual generalization. Using the syntactic aspect of language as a testbed, our analyses of 43 languages reveal a high degree of alignability among the spaces of structural concepts within each language for both encoder-only and decoder-only LLMs. We then propose a meta-learning-based method to learn to align conceptual spaces of different languages, which facilitates zero-shot and few-shot generalization in concept classification and also offers insights into the cross-lingual in-context learning phenomenon. Experiments on syntactic analysis tasks show that our approach achieves competitive results with state-of-the-art methods and narrows the performance gap between languages, particularly benefiting those with limited resources.

Title: Asymptotic Exit Location Distributions in the Stochastic Exit Problem

Abstract: Consider a two-dimensional continuous-time dynamical system, with an attracting fixed point $S$. If the deterministic dynamics are perturbed by white noise (random perturbations) of strength $\epsilon$, the system state will eventually leave the domain of attraction $\Omega$ of $S$. We analyse the case when, as $\epsilon\to0$, the exit location on the boundary $\partial\Omega$ is increasingly concentrated near a saddle point $H$ of the deterministic dynamics. We show that the asymptotic form of the exit location distribution on $\partial\Omega$ is generically non-Gaussian and asymmetric, and classify the possible limiting distributions. A key role is played by a parameter $\mu$, equal to the ratio $|\lambda_s(H)|/\lambda_u(H)$ of the stable and unstable eigenvalues of the linearized deterministic flow at $H$. If $\mu<1$ then the exit location distribution is generically asymptotic as $\epsilon\to0$ to a Weibull distribution with shape parameter $2/\mu$, on the $O(\epsilon^{\mu/2})$ length scale near $H$. If $\mu>1$ it is generically asymptotic to a distribution on the $O(\epsilon^{1/2})$ length scale, whose moments we compute. The asymmetry of the asymptotic exit location distribution is attributable to the generic presence of a `classically forbidden' region: a wedge-shaped subset of $\Omega$ with $H$ as vertex, which is reached from $S$, in the $\epsilon\to0$ limit, only via `bent' (non-smooth) fluctuational paths that first pass through the vicinity of $H$. We deduce from the presence of this forbidden region that the classical Eyring formula for the small-$\epsilon$ exponential asymptotics of the mean first exit time is generically inapplicable.


are all naruto shippuden episodes dubbed yet?

Scifi Stackexchange

Q: What is the only Jutsu included in the manga from Filler Episodes of Naruto Shippuden Anime My friend challenged me with this question a while ago.
Which jutsu premiered in Naruto Shippuden, but was thereafter picked up by Masashi Kishimoto into his Manga?
This has apparently only happened once, all the other jutsu's happened the other way.  He created them and put them into the Manga, then were disseminated into the Shippuden

A: The only jutsu that matches somehow the requirements (at least to my knowledge and search) is the Water Release: Water Shuriken (Suiton: Mizu Shuriken):

This jutsu appeared first in Naruto: Shippūden #412 - Neji's Judgment which aired in Japan on May 14, 2015 and was later also used in the Naruto novel Gaara Hiden: A Sandstorm Mirage which was published in Japan on June 4, 2015.
However this is a novel and not the original manga although is was illustrated by Masashi Kishimoto.


what is the difference between phd and pdf?

Stackoverflow Stackexchange

Q: Importing Drupal 5 content into Drupal 6 I need to import several thousand nodes from a Drupal 5 sql dump into a Drupal 6 website. I tried using Node Import and Import/Export API but very little success... any ideas as to how this can be done?
Many thanks!
Jane

A: I used Views Bonus Pack on a Drupal 5 site to export the nodes into a csv file.
Then I used the feeds module to import the csv into a Drupal 6 site.

Math Stackexchange

Q: Difference between a PDF and PMF What is the difference between a probability density function and a probability mass function?

A: One is for discrete variables, the other for continuous.
They are also interpreted differently. The PDF is a probability density. If f(x) is a PDF, f(x) doesn't tell you the probability of getting x (In fact, the probability of getting precisely x is 0).
The way you use a PDF is through integration. So, the probability of getting a value that is in a dx area around x is approximately f(x)*dx.


Title: Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

Abstract: We present Unified-IO 2, the first autoregressive multimodal model that is capable of understanding and generating image, text, audio, and action. To unify different modalities, we tokenize inputs and outputs -- images, text, audio, action, bounding boxes, etc., into a shared semantic space and then process them with a single encoder-decoder transformer model. Since training with such diverse modalities is challenging, we propose various architectural improvements to stabilize model training. We train our model from scratch on a large multimodal pre-training corpus from diverse sources with a multimodal mixture of denoisers objective. To learn an expansive set of skills, such as following multimodal instructions, we construct and finetune on an ensemble of 120 datasets with prompts and augmentations. With a single unified model, Unified-IO 2 achieves state-of-the-art performance on the GRIT benchmark and strong results in more than 35 benchmarks, including image generation and understanding, natural language understanding, video and audio understanding, and robotic manipulation. We release all our models to the research community.

Title: A Unified Programming Model for Heterogeneous Computing with CPU and Accelerator Technologies

Abstract: This paper consists of three parts. The first part provides a unified programming model for heterogeneous computing with CPU and accelerator (like GPU, FPGA, Google TPU, Atos QPU, and more) technologies. To some extent, this new programming model makes programming across CPUs and accelerators turn into usual programming tasks with common programming languages, and relieves complexity of programming across CPUs and accelerators. It can be achieved by extending file managements in common programming languages, such as C/C++, Fortran, Python, MPI, etc., to cover accelerators as I/O devices. In the second part, we show that all types of computer systems can be reduced to the simplest type of computer system, a single-core CPU computer system with I/O devices, by the unified programming model. Thereby, the unified programming model can truly build the programming of various computer systems on one API (i.e. file managements of common programming languages), and can make programming for various computer systems easier. In third part, we present a new approach to coupled applications computing (like multidisciplinary simulations) by the unified programming model. The unified programming model makes coupled applications computing more natural and easier since it only relies on its own power to couple multiple applications through MPI.


Good benchmark for multitask performance of llms

Title: Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?

Abstract: Large language models (LLMs) are typically prompted to follow a single instruction per inference call. In this work, we analyze whether LLMs also hold the capability to handle multiple instructions simultaneously, denoted as Multi-Task Inference. For this purpose, we introduce the MTI Bench(Multi-Task Inference Benchmark), a comprehensive evaluation benchmark encompassing 5,000 instances across 25 tasks. Each task in the MTI Bench involves 2 to 3 sub-tasks. As expected, we first demonstrate that Multi-Task Inference reduces the total inference time by 1.46 times in average since it does not require multiple inference calls. Interestingly, contrary to the expectation that LLMs would perform better when tasks are divided, we find that state-of-the-art LLMs, such as Llama-2-Chat-70B and GPT-4, show up to 7.3% and 12.4% improved performance with Multi-Task Inference compared to Single-Task Inference on the MTI Bench. We release the MTI Bench dataset and our code at this link https://github.com/guijinSON/MTI-Bench.

Title: Limit theorems and ergodicity for general bootstrap random walks

Abstract: Given the increments of a simple symmetric random walk $(X_n)_{n\ge0}$, we characterize all possible ways of recycling these increments into a simple symmetric random walk $(Y_n)_{n\ge0}$ adapted to the filtration of $(X_n)_{n\ge0}$. We study the long term behavior of a suitably normalized two-dimensional process $((X_n,Y_n))_{n\ge0}$. In particular, we provide necessary and sufficient conditions for the process to converge to a two-dimensional Brownian motion (possibly degenerate). We also discuss cases in which the limit is not Gaussian. Finally, we provide a simple necessary and sufficient condition for the ergodicity of the recycling transformation, thus generalizing results from Dubins and Smorodinsky (1992) and Fujita (2008), and solving the discrete version of the open problem of the ergodicity of the general L\'evy transformation (see Mansuy and Yor, 2006).


A paper evaluating whether updating one fact in a language model affects related facts.

Title: Evaluating the Ripple Effects of Knowledge Editing in Language Models

Abstract: Modern language models capture a large body of factual knowledge. However, some facts can be incorrectly induced or become obsolete over time, resulting in factually incorrect generations. This has led to the development of various editing methods that allow updating facts encoded by the model. Evaluation of these methods has primarily focused on testing whether an individual fact has been successfully injected, and if similar predictions for other subjects have not changed. Here we argue that such evaluation is limited, since injecting one fact (e.g. ``Jack Depp is the son of Johnny Depp'') introduces a ``ripple effect'' in the form of additional facts that the model needs to update (e.g.``Jack Depp is the sibling of Lily-Rose Depp''). To address this issue, we propose a novel set of evaluation criteria that consider the implications of an edit on related facts. Using these criteria, we then construct RippleEdits, a diagnostic benchmark of 5K factual edits, capturing a variety of types of ripple effects. We evaluate prominent editing methods on RippleEdits, showing that current methods fail to introduce consistent changes in the model's knowledge. In addition, we find that a simple in-context editing baseline obtains the best scores on our benchmark, suggesting a promising research direction for model editing.

Title: WikiFactDiff: A Large, Realistic, and Temporally Adaptable Dataset for Atomic Factual Knowledge Update in Causal Language Models

Abstract: The factuality of large language model (LLMs) tends to decay over time since events posterior to their training are "unknown" to them. One way to keep models up-to-date could be factual update: the task of inserting, replacing, or removing certain simple (atomic) facts within the model. To study this task, we present WikiFactDiff, a dataset that describes the evolution of factual knowledge between two dates as a collection of simple facts divided into three categories: new, obsolete, and static. We describe several update scenarios arising from various combinations of these three types of basic update. The facts are represented by subject-relation-object triples; indeed, WikiFactDiff was constructed by comparing the state of the Wikidata knowledge base at 4 January 2021 and 27 February 2023. Those fact are accompanied by verbalization templates and cloze tests that enable running update algorithms and their evaluation metrics. Contrary to other datasets, such as zsRE and CounterFact, WikiFactDiff constitutes a realistic update setting that involves various update scenarios, including replacements, archival, and new entity insertions. We also present an evaluation of existing update algorithms on WikiFactDiff.


A method using descriptors from large language models to improve accuracy and mitigate bias in visual classification tasks.

Title: Images in Language Space: Exploring the Suitability of Large Language Models for Vision & Language Tasks

Abstract: Large language models have demonstrated robust performance on various language tasks using zero-shot or few-shot learning paradigms. While being actively researched, multimodal models that can additionally handle images as input have yet to catch up in size and generality with language-only models. In this work, we ask whether language-only models can be utilised for tasks that require visual input -- but also, as we argue, often require a strong reasoning component. Similar to some recent related work, we make visual information accessible to the language model using separate verbalisation models. Specifically, we investigate the performance of open-source, open-access language models against GPT-3 on five vision-language tasks when given textually-encoded visual information. Our results suggest that language models are effective for solving vision-language tasks even with limited samples. This approach also enhances the interpretability of a model's output by providing a means of tracing the output back through the verbalised image content.

Title: Visual Classification via Description from Large Language Models

Abstract: Vision-language models (VLMs) such as CLIP have shown promising performance on a variety of recognition tasks using the standard zero-shot classification procedure -- computing similarity between the query image and the embedded words for each category. By only using the category name, they neglect to make use of the rich context of additional information that language affords. The procedure gives no intermediate understanding of why a category is chosen, and furthermore provides no mechanism for adjusting the criteria used towards this decision. We present an alternative framework for classification with VLMs, which we call classification by description. We ask VLMs to check for descriptive features rather than broad categories: to find a tiger, look for its stripes; its claws; and more. By basing decisions on these descriptors, we can provide additional cues that encourage using the features we want to be used. In the process, we can get a clear idea of what features the model uses to construct its decision; it gains some level of inherent explainability. We query large language models (e.g., GPT-3) for these descriptors to obtain them in a scalable way. Extensive experiments show our framework has numerous advantages past interpretability. We show improvements in accuracy on ImageNet across distribution shifts; demonstrate the ability to adapt VLMs to recognize concepts unseen during training; and illustrate how descriptors can be edited to effectively mitigate bias compared to the baseline.


Highest BLEU scores for fully NAT in WMT14 En-De with faster inference.

Title: Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade

Abstract: Fully non-autoregressive neural machine translation (NAT) is proposed to simultaneously predict tokens with single forward of neural networks, which significantly reduces the inference latency at the expense of quality drop compared to the Transformer baseline. In this work, we target on closing the performance gap while maintaining the latency advantage. We first inspect the fundamental issues of fully NAT models, and adopt dependency reduction in the learning space of output tokens as the basic guidance. Then, we revisit methods in four different aspects that have been proven effective for improving NAT models, and carefully combine these techniques with necessary modifications. Our extensive experiments on three translation benchmarks show that the proposed system achieves the new state-of-the-art results for fully NAT models, and obtains comparable performance with the autoregressive and iterative NAT systems. For instance, one of the proposed models achieves 27.49 BLEU points on WMT14 En-De with approximately 16.5X speed up at inference time.


A comprehensive analysis of a 280 billion parameter language model and its performance across 152 diverse tasks

Title: Baichuan 2: Open Large-scale Language Models

Abstract: Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering. However, most powerful LLMs are closed-source or limited in their capability for languages other than English. In this technical report, we present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan 2 excels in vertical domains such as medicine and law. We will release all pre-training model checkpoints to benefit the research community in better understanding the training dynamics of Baichuan 2.

Title: Seq2Mol: Automatic design of de novo molecules conditioned by the target protein sequences through deep neural networks

Abstract: De novo design of molecules has recently enjoyed the power of generative deep neural networks. Current approaches aim to generate molecules either resembling the properties of the molecules of the training set or molecules that are optimized with respect to specific physicochemical properties. None of the methods generates molecules specific to a target protein. In the approach presented here, we introduce a method which is conditioned on the protein target sequence to generate de novo molecules that are relevant to the target. We use an implementation adapted from Google's "Show and Tell" image caption generation method, to generate SMILES strings of molecules from protein sequence embeddings generated by a deep bi-directional language model ELMo. ELMo is used to generate contextualized embedding vectors of the protein sequence. Using reinforcement learning, the trained model is further optimized through augmented episodic likelihood to increase the diversity of the generated compounds compared to the training set. We used the model to generate compounds for two major drug target families, i.e. for GPCRs and Tyrosine Kinase targets. The model generated compounds which are structurally different form the training set, while also being more similar to compounds known to bind to the two families of drug targets compared to a random set of molecules. The compounds further display reasonable synthesizability and drug-likeness scores.


Paper introducing a model connecting object segmentation masks with natural language responses in multimodal settings.

Title: Dynamic Multimodal Instance Segmentation guided by natural language queries

Abstract: We address the problem of segmenting an object given a natural language expression that describes it. Current techniques tackle this task by either (\textit{i}) directly or recursively merging linguistic and visual information in the channel dimension and then performing convolutions; or by (\textit{ii}) mapping the expression to a space in which it can be thought of as a filter, whose response is directly related to the presence of the object at a given spatial coordinate in the image, so that a convolution can be applied to look for the object. We propose a novel method that integrates these two insights in order to fully exploit the recursive nature of language. Additionally, during the upsampling process, we take advantage of the intermediate information generated when downsampling the image, so that detailed segmentations can be obtained. We compare our method against the state-of-the-art approaches in four standard datasets, in which it surpasses all previous methods in six of eight of the splits for this task.

Title: Comprehensive Multi-Modal Interactions for Referring Image Segmentation

Abstract: We investigate Referring Image Segmentation (RIS), which outputs a segmentation map corresponding to the natural language description. Addressing RIS efficiently requires considering the interactions happening across visual and linguistic modalities and the interactions within each modality. Existing methods are limited because they either compute different forms of interactions sequentially (leading to error propagation) or ignore intramodal interactions. We address this limitation by performing all three interactions simultaneously through a Synchronous Multi-Modal Fusion Module (SFM). Moreover, to produce refined segmentation masks, we propose a novel Hierarchical Cross-Modal Aggregation Module (HCAM), where linguistic features facilitate the exchange of contextual information across the visual hierarchy. We present thorough ablation studies and validate our approach's performance on four benchmark datasets, showing considerable performance gains over the existing state-of-the-art (SOTA) methods.


Stats Stackexchange

Q: Should I treat age as a continuous variable or factor? Age is a predictor variable among a few others (gender, weight and height) and my response is the log-odds of a certain disease (binomial glm).
My age data runs from 21 until 40. I am not sure whether to treat age as a continuous variable or as a factor with age groups: 21-25, 26-30, 31-35, 36-40.
Are there any plots which I can produce that can help determine which would be the better approach?

A: It depends on the context.
For example if you are looking for the effect of age on children's height, it makes sense to look at it as a continuous ( integer) value.
If you're looking for e.g. the effect of age on oncogenesis then it makes sense if you look at age groups. Young vs old, above 55 and below 55, ...
For your example, unless age is a confounder of a hidden factor such as for example being college grad or still a student ( risk factor for young adults STD infection), I'd bin my data into reasonable bin sizes.

Stats Stackexchange

Q: Are the variable types here considered correct? If we want to determine the variable types, will it be as follows for the below variables?
Age ---> quantitative, discrete (we can count)
Fitness ---> If the values that we will enter here are 0 and 1 only, will the type of this variable be qualitative, nominal?
Thanks a lot.

A: Questions like this expose the problems with Stevens' typology of variables. Some variables just aren't any of his categories; or are hard to categorize. Age is ratio level data in the sense that someone who is (say) 40 years old is twice as old as someone who is 20 years old. You just haven't got it measured precisely. But so? What difference does that make?
Dichotomies - here it really doesn't matter HOW you treat them for most purposes. E.g. in regression, you can consider them to be nominal, ordinal or interval and you will get the same results.
I wrote about other problems with this typology on my blog


A framework to improve logical problem-solving in language models integrating symbolic formulation and self-refinement.

Title: On the three state Potts model with competing interactions on the Bethe lattice

Abstract: In the present paper the three state Potts model with competing binary interactions (with couplings $J$ and $J_p$) on the second order Bethe lattice is considered. The recurrent equations for the partition functions are derived. When $J_p=0$, by means of a construction of a special class of limiting Gibbs measures, it is shown how these equations are related with the surface energy of the Hamiltonian. This relation reduces the problem of describing the limit Gibbs measures to find of solutions of a nonlinear functional equation. Moreover, the set of ground states of the one-level model is completely described. Using this fact, one finds Gibbs measures (pure phases) associated with the translation-invariant ground states. The critical temperature is exactly found and the phase diagram is presented. The free energies corresponding to translations-invariant Gibbs measures are found. Certain physical quantities are calculated as well.

Title: Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning

Abstract: Large Language Models (LLMs) have shown human-like reasoning abilities but still struggle with complex logical problems. This paper introduces a novel framework, Logic-LM, which integrates LLMs with symbolic solvers to improve logical problem-solving. Our method first utilizes LLMs to translate a natural language problem into a symbolic formulation. Afterward, a deterministic symbolic solver performs inference on the formulated problem. We also introduce a self-refinement module, which utilizes the symbolic solver's error messages to revise symbolic formalizations. We demonstrate Logic-LM's effectiveness on five logical reasoning datasets: ProofWriter, PrOntoQA, FOLIO, LogicalDeduction, and AR-LSAT. On average, Logic-LM achieves a significant performance boost of 39.2% over using LLM alone with standard prompting and 18.4% over LLM with chain-of-thought prompting. Our findings suggest that Logic-LM, by combining LLMs with symbolic logic, offers a promising avenue for faithful logical reasoning. Code and data are publicly available at https://github.com/teacherpeterpan/Logic-LLM.


Looking for the paper on the 2020 CLEF edition covering automatic claim verification in social media.

Title: Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media

Abstract: We present an overview of the third edition of the CheckThat! Lab at CLEF 2020. The lab featured five tasks in two different languages: English and Arabic. The first four tasks compose the full pipeline of claim verification in social media: Task 1 on check-worthiness estimation, Task 2 on retrieving previously fact-checked claims, Task 3 on evidence retrieval, and Task 4 on claim verification. The lab is completed with Task 5 on check-worthiness estimation in political debates and speeches. A total of 67 teams registered to participate in the lab (up from 47 at CLEF 2019), and 23 of them actually submitted runs (compared to 14 at CLEF 2019). Most teams used deep neural networks based on BERT, LSTMs, or CNNs, and achieved sizable improvements over the baselines on all tasks. Here we describe the tasks setup, the evaluation results, and a summary of the approaches used by the participants, and we discuss some lessons learned. Last but not least, we release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in the important tasks of check-worthiness estimation and automatic claim verification.


Stats Stackexchange

Q: Is it a problem to get a negative adjusted r-squared? Background:
I have the cross-sectional model:
$Y_{i}  = a + b X_{1,i} + c X_{2,i} + d X_{3,i} + e X_{4,i} + \nu_i$.
The application is corporate finance. So each $Y_i$ is something like the change in return on assets over a 1 year period for firm $i$, and the regressors are typical corporate finance variables.
In corporate finance, very small values of $R^2$ are common, even sometimes $1\%$. My $R^2$ is around $1\%$ but my Adjusted $R^2$ is $-0.2\%$.
I have never seen papers report negative Adjusted $R^2$ but this could just be because they omit the publication of their adjusted $R^2$ when they see that it is negative.
Question
Is there some problem when the adjusted $R^2$ is negative?

A: The formula for adjusted R square allows it to be negative.  It is intended to approximate the actual percentage variance explained.  So if the actual R square is close to zero the adjusted R square can be slightly negative.  Just think of it as an estimate of zero.


Paper showcasing reinforcement learning from human feedback to train beneficial and safe AI assistants.

Title: Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Abstract: We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.


Looking for a paper on Fine-grained Entity Type Classification using a neural network with hierarchy-aware loss.

Title: Graph Neural Network Approach to Semantic Type Detection in Tables

Abstract: This study addresses the challenge of detecting semantic column types in relational tables, a key task in many real-world applications. While language models like BERT have improved prediction accuracy, their token input constraints limit the simultaneous processing of intra-table and inter-table information. We propose a novel approach using Graph Neural Networks (GNNs) to model intra-table dependencies, allowing language models to focus on inter-table information. Our proposed method not only outperforms existing state-of-the-art algorithms but also offers novel insights into the utility and functionality of various GNN types for semantic type detection. The code is available at https://github.com/hoseinzadeehsan/GAIT

Title: Neural Fine-Grained Entity Type Classification with Hierarchy-Aware Loss

Abstract: The task of Fine-grained Entity Type Classification (FETC) consists of assigning types from a hierarchy to entity mentions in text. Existing methods rely on distant supervision and are thus susceptible to noisy labels that can be out-of-context or overly-specific for the training sentence. Previous methods that attempt to address these issues do so with heuristics or with the help of hand-crafted features. Instead, we propose an end-to-end solution with a neural network model that uses a variant of cross- entropy loss function to handle out-of-context labels, and hierarchical loss normalization to cope with overly-specific ones. Also, previous work solve FETC a multi-label classification followed by ad-hoc post-processing. In contrast, our solution is more elegant: we use public word embeddings to train a single-label that jointly learns representations for entity mentions and their context. We show experimentally that our approach is robust against noise and consistently outperforms the state-of-the-art on established benchmarks for the task.


A paper introducing a new large language model

Title: Larger-Context Language Modelling

Abstract: In this work, we propose a novel method to incorporate corpus-level discourse information into language modelling. We call this larger-context language model. We introduce a late fusion approach to a recurrent language model based on long short-term memory units (LSTM), which helps the LSTM unit keep intra-sentence dependencies and inter-sentence dependencies separate from each other. Through the evaluation on three corpora (IMDB, BBC, and PennTree Bank), we demon- strate that the proposed model improves perplexity significantly. In the experi- ments, we evaluate the proposed approach while varying the number of context sentences and observe that the proposed late fusion is superior to the usual way of incorporating additional inputs to the LSTM. By analyzing the trained larger- context language model, we discover that content words, including nouns, adjec- tives and verbs, benefit most from an increasing number of context sentences. This analysis suggests that larger-context language model improves the unconditional language model by capturing the theme of a document better and more easily.

Title: Large Language Models: A Survey

Abstract: Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws \cite{kaplan2020scaling,hoffmann2022training}. The research area of LLMs, while very recent, is evolving rapidly in many different ways. In this paper, we review some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discuss their characteristics, contributions and limitations. We also give an overview of techniques developed to build, and augment LLMs. We then survey popular datasets prepared for LLM training, fine-tuning, and evaluation, review widely used LLM evaluation metrics, and compare the performance of several popular LLMs on a set of representative benchmarks. Finally, we conclude the paper by discussing open challenges and future research directions.


How can BERT models be adapted to perform better in biomedical natural language processing?

Title: Spatiotemporal complexity of a ratio-dependent predator-prey system

Abstract: In this paper, we investigate the emergence of a ratio-dependent predator-prey system with Michaelis-Menten-type functional response and reaction-diffusion. We derive the conditions for Hopf, Turing and Wave bifurcation on a spatial domain. Furthermore, we present a theoretical analysis of evolutionary processes that involves organisms distribution and their interaction of spatially distributed population with local diffusion. The results of numerical simulations reveal that the typical dynamics of population density variation is the formation of isolated groups, i.e., stripelike or spotted or coexistence of both. Our study shows that the spatially extended model has not only more complex dynamic patterns in the space, but also chaos and spiral waves. It may help us better understand the dynamics of an aquatic community in a real marine environment.

Title: Absorbing processes in Richardson diffusion: analytical results

Abstract: We consider the recently addressed problem of a passive particle (a predator), being the center of a ``sphere of interception'' of radius $R$ and able to absorb other passive particles (the preys) entering into the sphere. Assuming that all the particles are advected by a turbulent flow and that, in particular, the Richardson equation properly describes the relative dispersion, we calculate an analytical expression for the flux into the sphere as a function of time, assuming an initial constant density of preys outside the sphere. In the same framework, we show that the distribution of times of first passage into the sphere has a $t^{-5/2}$ power law tail, seen in contrast to the $t^{-3/2}$ appearing in standard 3D diffusion. We also discuss the correction due to the integral length scale on the results in the stationary case.


Best practices and taxonomy of data selection for efficient language model pre-training.

Title: How to Train Data-Efficient LLMs

Abstract: The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we propose Density sampling, which models the data distribution to select a diverse sample. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories. Coverage sampling can recover the performance of the full data, while models trained on Ask-LLM data consistently outperform full-data training -- even when we reject 90% of the original dataset, while converging up to 70% faster.

Title: A Survey on Data Selection for Language Models

Abstract: A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training. However, naively training a model on all available data may not be optimal (or feasible), as the quality of available text data can vary. Filtering out data can also decrease the carbon footprint and financial costs of training models by reducing the amount of training required. Data selection methods aim to determine which candidate data points to include in the training dataset and how to appropriately sample from the selected data points. The promise of improved data selection methods has caused the volume of research in the area to rapidly expand. However, because deep learning is mostly driven by empirical evidence and experimentation on large-scale data is expensive, few organizations have the resources for extensive data selection research. Consequently, knowledge of effective data selection practices has become concentrated within a few organizations, many of which do not openly share their findings and methodologies. To narrow this gap in knowledge, we present a comprehensive review of existing literature on data selection methods and related research areas, providing a taxonomy of existing approaches. By describing the current landscape of research, this work aims to accelerate progress in data selection by establishing an entry point for new and established researchers. Additionally, throughout this review we draw attention to noticeable holes in the literature and conclude the paper by proposing promising avenues for future research.


Paper detailing advancements in self-supervised learning with ResNets that outperforms supervised benchmarks using ReLICv2.

Title: Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?

Abstract: Despite recent progress made by self-supervised methods in representation learning with residual networks, they still underperform supervised learning on the ImageNet classification benchmark, limiting their applicability in performance-critical settings. Building on prior theoretical insights from ReLIC [Mitrovic et al., 2021], we include additional inductive biases into self-supervised learning. We propose a new self-supervised representation learning method, ReLICv2, which combines an explicit invariance loss with a contrastive objective over a varied set of appropriately constructed data views to avoid learning spurious correlations and obtain more informative representations. ReLICv2 achieves $77.1\%$ top-$1$ accuracy on ImageNet under linear evaluation on a ResNet50, thus improving the previous state-of-the-art by absolute $+1.5\%$; on larger ResNet models, ReLICv2 achieves up to $80.6\%$ outperforming previous self-supervised approaches with margins up to $+2.3\%$. Most notably, ReLICv2 is the first unsupervised representation learning method to consistently outperform the supervised baseline in a like-for-like comparison over a range of ResNet architectures. Using ReLICv2, we also learn more robust and transferable representations that generalize better out-of-distribution than previous work, both on image classification and semantic segmentation. Finally, we show that despite using ResNet encoders, ReLICv2 is comparable to state-of-the-art self-supervised vision transformers.

Title: A Recharge Oscillator Model for Interannual Variability in Venus' Clouds

Abstract: Sulfur dioxide is a radiatively and chemically important trace gas in the atmosphere of Venus and its abundance at the cloud-tops has been observed to vary on interannual to decadal timescales. This variability is thought to come from changes in the strength of convection which transports sulfur dioxide to the cloud-tops, {although} the dynamics behind such convective variability are unknown. Here we propose a new conceptual model for convective variability that links the radiative effects of water abundance at the cloud-base to convective strength within the clouds, which in turn affects water transport within the cloud. The model consists of two coupled equations which are identified as a recharge-discharge oscillator. The solutions of the coupled equations are finite amplitude sustained oscillations in convective strength and cloud-base water abundance on 3-9 year timescales. The characteristic oscillation timescale is given by the geometric mean of the radiative cooling time and the eddy mixing time near the base of the convective clouds.


Open-source toolkit for training custom models for NLP tasks like translation and summarization using PyTorch

Title: fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Abstract: fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs. A demo video can be found at https://www.youtube.com/watch?v=OtgDdWtHvto

Title: OpenNMT: Open-Source Toolkit for Neural Machine Translation

Abstract: We describe an open-source toolkit for neural machine translation (NMT). The toolkit prioritizes efficiency, modularity, and extensibility with the goal of supporting NMT research into model architectures, feature representations, and source modalities, while maintaining competitive performance and reasonable training requirements. The toolkit consists of modeling and translation support, as well as detailed pedagogical documentation about the underlying techniques.


Research on leveraging knowledge graphs in protein pretraining models to enhance performance in protein-protein interaction and function prediction.

Title: OntoProtein: Protein Pretraining With Gene Ontology Embedding

Abstract: Self-supervised protein language models have proved their effectiveness in learning the proteins representations. With the increasing computational power, current protein language models pre-trained with millions of diverse sequences can advance the parameter scale from million-level to billion-level and achieve remarkable improvement. However, those prevailing approaches rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better protein representations. We argue that informative biology knowledge in KGs can enhance protein representation with external knowledge. In this work, we propose OntoProtein, the first general framework that makes use of structure in GO (Gene Ontology) into protein pre-training models. We construct a novel large-scale knowledge graph that consists of GO and its related proteins, and gene annotation texts or protein sequences describe all nodes in the graph. We propose novel contrastive learning with knowledge-aware negative sampling to jointly optimize the knowledge graph and protein embedding during pre-training. Experimental results show that OntoProtein can surpass state-of-the-art methods with pre-trained protein language models in TAPE benchmark and yield better performance compared with baselines in protein-protein interaction and protein function prediction. Code and datasets are available in https://github.com/zjunlp/OntoProtein.


Looking for a research paper that introduces a new method using text-guided attention to improve polyp segmentation in colonoscopy images.

Title: TGANet: Text-guided attention for improved polyp segmentation

Abstract: Colonoscopy is a gold standard procedure but is highly operator-dependent. Automated polyp segmentation, a precancerous precursor, can minimize missed rates and timely treatment of colon cancer at an early stage. Even though there are deep learning methods developed for this task, variability in polyp size can impact model training, thereby limiting it to the size attribute of the majority of samples in the training dataset that may provide sub-optimal results to differently sized polyps. In this work, we exploit size-related and polyp number-related features in the form of text attention during training. We introduce an auxiliary classification task to weight the text-based embedding that allows network to learn additional feature representations that can distinctly adapt to differently sized polyps and can adapt to cases with multiple polyps. Our experimental results demonstrate that these added text embeddings improve the overall performance of the model compared to state-of-the-art segmentation methods. We explore four different datasets and provide insights for size-specific improvements. Our proposed text-guided attention network (TGANet) can generalize well to variable-sized polyps in different datasets.

Title: The W_t Transcendental Function and Quantum Mechanical Applications

Abstract: We discuss the function wt(x) defined via the implicit equation wt(x)*tan[wt(x)]=x which appears in certain quantum mechanical and field theoretic applications. We investigate its analytic structure, develop series expansions for both small and large x, and provide various techniques for its numerical evaluation in the complex plane.


How can we improve the interpretability of deep learning models using simpler models?

Title: A Detailed Study of Interpretability of Deep Neural Network based Top Taggers

Abstract: Recent developments in the methods of explainable AI (XAI) allow researchers to explore the inner workings of deep neural networks (DNNs), revealing crucial information about input-output relationships and realizing how data connects with machine learning models. In this paper we explore interpretability of DNN models designed to identify jets coming from top quark decay in high energy proton-proton collisions at the Large Hadron Collider (LHC). We review a subset of existing top tagger models and explore different quantitative methods to identify which features play the most important roles in identifying the top jets. We also investigate how and why feature importance varies across different XAI metrics, how correlations among features impact their explainability, and how latent space representations encode information as well as correlate with physically meaningful quantities. Our studies uncover some major pitfalls of existing XAI methods and illustrate how they can be overcome to obtain consistent and meaningful interpretation of these models. We additionally illustrate the activity of hidden layers as Neural Activation Pattern (NAP) diagrams and demonstrate how they can be used to understand how DNNs relay information across the layers and how this understanding can help to make such models significantly simpler by allowing effective model reoptimization and hyperparameter tuning. These studies not only facilitate a methodological approach to interpreting models but also unveil new insights about what these models learn. Incorporating these observations into augmented model design, we propose the Particle Flow Interaction Network (PFIN) model and demonstrate how interpretability-inspired model augmentation can improve top tagging performance.

Title: Interpretability Illusions in the Generalization of Simplified Models

Abstract: A common method to study deep learning systems is to use simplified model representations--for example, using singular value decomposition to visualize the model's hidden states in a lower dimensional space. This approach assumes that the results of these simplifications are faithful to the original model. Here, we illustrate an important caveat to this assumption: even if the simplified representations can accurately approximate the full model on the training set, they may fail to accurately capture the model's behavior out of distribution. We illustrate this by training Transformer models on controlled datasets with systematic generalization splits, including the Dyck balanced-parenthesis languages and a code completion task. We simplify these models using tools like dimensionality reduction and clustering, and then explicitly test how these simplified proxies match the behavior of the original model. We find consistent generalization gaps: cases in which the simplified proxies are more faithful to the original model on the in-distribution evaluations and less faithful on various tests of systematic generalization. This includes cases where the original model generalizes systematically but the simplified proxies fail, and cases where the simplified proxies generalize better. Together, our results raise questions about the extent to which mechanistic interpretations derived using tools like SVD can reliably predict what a model will do in novel situations.


is bloodborne in the same universe as dark souls?

Math Stackexchange

Q: Solving $\arcsin(1-x)-2\arcsin(x)=\pi/2$ \begin{eqnarray*}
\arcsin(1-x)-2\arcsin(x) & = & \frac{\pi}{2}\\
1-x & = & \sin\left(\frac{\pi}{2}+2\arcsin(x)\right)\\
 & = & \cos\left(2\arcsin(x)\right)\\
 & = & 1-2\left(\sin\left(\arcsin(x)\right)\right)^{2}\\
 & = & 1-2x^{2}\\
x & = & 2x^{2}\\
x\left(x-\frac{1}{2}\right) & = & 0
\end{eqnarray*}
So $x=0$ or $x=\frac{1}{2}$
But puttig $x=\frac{1}{2}$ in the original expression gives $-\frac {\pi} 4 \ne \frac \pi 2$
So, why do we get $x=-1/2$ as an answer?

A: In your first step you added an extra solution.
Since $\arcsin x$ must be smaller than $\pi/2$, the first line reads:
$$\arcsin(1-x)= \frac{\pi}{2}+2\arcsin(x) \le \frac{\pi}{2}$$
Thus, $x\le 0$ as well.
Now, by taking the $\sin$ of both sides, you took a function that was only defined up to $x=1$ (e.g. $\arcsin(x-1)$ ) and extended it to all reals (e.g $x-1$). Here is where you added the extra solution.

Stackoverflow Stackexchange

Q: Stop UIAlertView from dismissing Can you stop UIAlertView from dismissing?
I want based on what button is pressed, to either animate adding a UITextField, or dismiss it. But I can't add a text field after the alert is dismissed.

A: You should implement the UIAlertViewDelegate method :
- (void)alertView:(UIAlertView *)alertView willDismissWithButtonIndex:(NSInteger)buttonIndex

and based on the button index you can perform your custom action.
Otherwise you should either create your own alertview class or subclass UIAlertView.
If you choose to subclass you can override the method :
-(void)dismissWithClickedButtonIndex:(NSInteger)buttonIndex animated:(BOOL)animated

to avoid dismissing the alertView if a specific button is pressed


How well do ChatGPT and GPT-4 perform on logical reasoning datasets?

Title: Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

Abstract: Harnessing logical reasoning ability is a comprehensive natural language understanding endeavor. With the release of Generative Pretrained Transformer 4 (GPT-4), highlighted as "advanced" at reasoning tasks, we are eager to learn the GPT-4 performance on various logical reasoning tasks. This report analyses multiple logical reasoning datasets, with popular benchmarks like LogiQA and ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. We further construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4. We also make a performance comparison between ChatGPT and GPT-4. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. With early access to the GPT-4 API we are able to conduct intense experiments on the GPT-4 model. The results show GPT-4 yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets. We release the prompt-style logical reasoning datasets as a benchmark suite and name it LogiEval.


Research on improving the performance and energy efficiency of neural networks by utilizing 8-bit precision.

Title: Understanding the Impact of Precision Quantization on the Accuracy and Energy of Neural Networks

Abstract: Deep neural networks are gaining in popularity as they are used to generate state-of-the-art results for a variety of computer vision and machine learning applications. At the same time, these networks have grown in depth and complexity in order to solve harder problems. Given the limitations in power budgets dedicated to these networks, the importance of low-power, low-memory solutions has been stressed in recent years. While a large number of dedicated hardware using different precisions has recently been proposed, there exists no comprehensive study of different bit precisions and arithmetic in both inputs and network parameters. In this work, we address this issue and perform a study of different bit-precisions in neural networks (from floating-point to fixed-point, powers of two, and binary). In our evaluation, we consider and analyze the effect of precision scaling on both network accuracy and hardware metrics including memory footprint, power and energy consumption, and design area. We also investigate training-time methodologies to compensate for the reduction in accuracy due to limited bit precision and demonstrate that in most cases, precision scaling can deliver significant benefits in design metrics at the cost of very modest decreases in network accuracy. In addition, we propose that a small portion of the benefits achieved when using lower precisions can be forfeited to increase the network size and therefore the accuracy. We evaluate our experiments, using three well-recognized networks and datasets to show its generality. We investigate the trade-offs and highlight the benefits of using lower precisions in terms of energy and memory footprint.

Title: Tartan: Accelerating Fully-Connected and Convolutional Layers in Deep Learning Networks by Exploiting Numerical Precision Variability

Abstract: Tartan (TRT), a hardware accelerator for inference with Deep Neural Networks (DNNs), is presented and evaluated on Convolutional Neural Networks. TRT exploits the variable per layer precision requirements of DNNs to deliver execution time that is proportional to the precision p in bits used per layer for convolutional and fully-connected layers. Prior art has demonstrated an accelerator with the same execution performance only for convolutional layers. Experiments on image classification CNNs show that on average across all networks studied, TRT outperforms a state-of-the-art bit-parallel accelerator by 1:90x without any loss in accuracy while it is 1:17x more energy efficient. TRT requires no network retraining while it enables trading off accuracy for additional improvements in execution performance and energy efficiency. For example, if a 1% relative loss in accuracy is acceptable, TRT is on average 2:04x faster and 1:25x more energy efficient than a conventional bit-parallel accelerator. A Tartan configuration that processes 2-bits at time, requires less area than the 1-bit configuration, improves efficiency to 1:24x over the bit-parallel baseline while being 73% faster for convolutional layers and 60% faster for fully-connected layers is also presented.


I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Efficient transformer models for multi-page document classification

Title: Revisiting Transformer-based Models for Long Document Classification

Abstract: The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph documents are common and they cannot be efficiently encoded by vanilla Transformer-based models. We compare different Transformer-based Long Document Classification (TrLDC) approaches that aim to mitigate the computational overhead of vanilla transformers to encode much longer text, namely sparse attention and hierarchical encoding methods. We examine several aspects of sparse attention (e.g., size of local attention window, use of global attention) and hierarchical (e.g., document splitting strategy) transformers on four document classification datasets covering different domains. We observe a clear benefit from being able to process longer text, and, based on our results, we derive practical advice of applying Transformer-based models on long document classification tasks.


How to combine GPT-2 and BERT for better question generation and answering?

Title: Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds

Abstract: Automatic question generation aims at the generation of questions from a context, with the corresponding answers being sub-spans of the given passage. Whereas, most of the methods mostly rely on heuristic rules to generate questions, more recently also neural network approaches have been proposed. In this work, we propose a variant of the self-attention Transformer network architectures model to generate meaningful and diverse questions. To this end, we propose an easy to use model consisting of the conjunction of the Transformer decoder GPT-2 model with Transformer encoder BERT for the downstream task for question answering. The model is trained in an end-to-end fashion, where the language model is trained to produce a question-answer-aware input representation that facilitates to generate an answer focused question. Our result of neural question generation from text on the SQuAD 1.1 dataset suggests that our method can produce semantically correct and diverse questions. Additionally, we assessed the performance of our proposed method for the downstream task of question answering. The analysis shows that our proposed generation & answering collaboration framework relatively improves both tasks and is particularly powerful in the semi-supervised setup. The results further suggest a robust and comparably lean pipeline facilitating question generation in the small-data regime.

Title: Radiation flux and spectrum in the Vaidya collapse model

Abstract: We consider the quantization of a massless scalar field, using the geometric optics approximation, in the background spacetime of a collapsing spherical self-similar Vaidya star, which forms a black hole or a naked singularity. We show that the outgoing radiation flux of the quantized scalar field diverges on the Cauchy horizon. The spectrum of the produced scalar partcles is non-thermal when the background develops a naked singularity. These results are analogous to those obtained for the scalar quantization on a self-similar dust cloud.


Paper introducing a transformer-based model trained on multiple visual modalities achieving high accuracy.

Title: Omnivore: A Single Model for Many Visual Modalities

Abstract: Prior work has studied different visual modalities in isolation and developed separate architectures for recognition of images, videos, and 3D data. Instead, in this paper, we propose a single model which excels at classifying images, videos, and single-view 3D data using exactly the same model parameters. Our 'Omnivore' model leverages the flexibility of transformer-based architectures and is trained jointly on classification tasks from different modalities. Omnivore is simple to train, uses off-the-shelf standard datasets, and performs at-par or better than modality-specific models of the same size. A single Omnivore model obtains 86.0% on ImageNet, 84.1% on Kinetics, and 67.1% on SUN RGB-D. After finetuning, our models outperform prior work on a variety of vision tasks and generalize across modalities. Omnivore's shared visual representation naturally enables cross-modal recognition without access to correspondences between modalities. We hope our results motivate researchers to model visual modalities together.

Title: Ab initio Molecular Dynamics Study of D_2 Desorption from Si(100)

Abstract: Ab initio molecular dynamics calculations of deuterium desorbing from Si(100) have been performed in order to monitor the energy redistribution among the hydrogen and silicon degrees of freedom during the desorption process. The calculations show that part of the potential energy at the transition state to desorption is transferred to the silicon lattice. The deuterium molecules leave the surface vibrationally hot and rotationally cold, in agreement with experiments; the mean kinetic energy, however, is larger than found in experiments.


A novel network combining convolution and transformer operations for better medical image segmentation.

Title: ConvFormer: Combining CNN and Transformer for Medical Image Segmentation

Abstract: Convolutional neural network (CNN) based methods have achieved great successes in medical image segmentation, but their capability to learn global representations is still limited due to using small effective receptive fields of convolution operations. Transformer based methods are capable of modelling long-range dependencies of information for capturing global representations, yet their ability to model local context is lacking. Integrating CNN and Transformer to learn both local and global representations while exploring multi-scale features is instrumental in further improving medical image segmentation. In this paper, we propose a hierarchical CNN and Transformer hybrid architecture, called ConvFormer, for medical image segmentation. ConvFormer is based on several simple yet effective designs. (1) A feed forward module of Deformable Transformer (DeTrans) is re-designed to introduce local information, called Enhanced DeTrans. (2) A residual-shaped hybrid stem based on a combination of convolutions and Enhanced DeTrans is developed to capture both local and global representations to enhance representation ability. (3) Our encoder utilizes the residual-shaped hybrid stem in a hierarchical manner to generate feature maps in different scales, and an additional Enhanced DeTrans encoder with residual connections is built to exploit multi-scale features with feature maps of different scales as input. Experiments on several datasets show that our ConvFormer, trained from scratch, outperforms various CNN- or Transformer-based architectures, achieving state-of-the-art performance.

Title: Rethinking Boundary Detection in Deep Learning Models for Medical Image Segmentation

Abstract: Medical image segmentation is a fundamental task in the community of medical image analysis. In this paper, a novel network architecture, referred to as Convolution, Transformer, and Operator (CTO), is proposed. CTO employs a combination of Convolutional Neural Networks (CNNs), Vision Transformer (ViT), and an explicit boundary detection operator to achieve high recognition accuracy while maintaining an optimal balance between accuracy and efficiency. The proposed CTO follows the standard encoder-decoder segmentation paradigm, where the encoder network incorporates a popular CNN backbone for capturing local semantic information, and a lightweight ViT assistant for integrating long-range dependencies. To enhance the learning capacity on boundary, a boundary-guided decoder network is proposed that uses a boundary mask obtained from a dedicated boundary detection operator as explicit supervision to guide the decoding learning process. The performance of the proposed method is evaluated on six challenging medical image segmentation datasets, demonstrating that CTO achieves state-of-the-art accuracy with a competitive model complexity.


How do different constraints in storytelling tasks impact the author's linguistic style?

Title: The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task

Abstract: A writer's style depends not just on personal traits but also on her intent and mental state. In this paper, we show how variants of the same writing task can lead to measurable differences in writing style. We present a case study based on the story cloze task (Mostafazadeh et al., 2016a), where annotators were assigned similar writing tasks with different constraints: (1) writing an entire story, (2) adding a story ending for a given story context, and (3) adding an incoherent ending to a story. We show that a simple linear classifier informed by stylistic features is able to successfully distinguish among the three cases, without even looking at the story context. In addition, combining our stylistic features with language model predictions reaches state of the art performance on the story cloze challenge. Our results demonstrate that different task framings can dramatically affect the way people write.

Title: Limits on dynamically generated spin-orbit coupling: Absence of $l=1$ Pomeranchuk instabilities in metals

Abstract: An ordered state in the spin sector that breaks parity without breaking time-reversal symmetry, i.e., that can be considered as dynamically generated spin-orbit coupling, was proposed to explain puzzling observations in a range of different systems. Here we derive severe restrictions for such a state that follow from a Ward identity related to spin conservation. It is shown that $l=1$ spin-Pomeranchuk instabilities are not possible in non-relativistic systems since the response of spin-current fluctuations is entirely incoherent and non-singular. This rules out relativistic spin-orbit coupling as an emergent low-energy phenomenon. We illustrate the exotic physical properties of the remaining higher angular momentum analogues of spin-orbit coupling and derive a geometric constraint for spin-orbit vectors in lattice systems.


Study on reducing gender bias in word-level models, showing improved results in bias evaluation metrics.

Title: Chaos or Noise - Difficulties of a Distinction

Abstract: In experiments, the dynamical behavior of systems is reflected in time series. Due to the finiteness of the observational data set it is not possible to reconstruct the invariant measure up to arbitrary fine resolution and arbitrary high embedding dimension. These restrictions limit our ability to distinguish between signals generated by different systems, such as regular, chaotic or stochastic ones, when analyzed from a time series point of view. We propose to classify the signal behavior, without referring to any specific model, as stochastic or deterministic on a certain scale of the resolution $\epsilon$, according to the dependence of the $(\epsilon,\tau)$-entropy, $h(\epsilon, \tau)$, and of the finite size Lyapunov exponent, $\lambda(\epsilon)$, on $\epsilon$.

Title: Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Function

Abstract: Gender bias exists in natural language datasets which neural language models tend to learn, resulting in biased text generation. In this research, we propose a debiasing approach based on the loss function modification. We introduce a new term to the loss function which attempts to equalize the probabilities of male and female words in the output. Using an array of bias evaluation metrics, we provide empirical evidence that our approach successfully mitigates gender bias in language models without increasing perplexity. In comparison to existing debiasing strategies, data augmentation, and word embedding debiasing, our method performs better in several aspects, especially in reducing gender bias in occupation words. Finally, we introduce a combination of data augmentation and our approach, and show that it outperforms existing strategies in all bias evaluation metrics.


Study analyzing the multitask accuracy of text models including GPT-3 on academic and professional knowledge areas.

Title: Vibrational Spectra of Defects in Silicon: An Orbital Radii Approach

Abstract: A phenomenological approach to the stretching mode vibrational frequencies of defects in semiconductors is proposed. A novel quantum scale is defined in terms of the first principles pseudopotential based orbital radius and the principal quantum number of the element concerned. A universal linear relationship between the Sanderson electronegativity and this quantum scale is established. Next, we show that the stretching mode vibrational frequencies of hydrogen and chlorine in the silicon network scale linearly with this quantum scale. Predictions and identifications of defect environments around the Si-H and Si-Cl are possible. The assignments of vibrational modes in porous silicon are critically examined. We discuss our proposed scale in the context of Mendeleveyan scales in general, and suggest justifications for it. We believe that our approach can be gainfully extended to the vibrational spectra of other semiconductors.

Title: Measuring Massive Multitask Language Understanding

Abstract: We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.


Comparison of sparse attention and hierarchical encoding in long document transformers


Paper introducing universal adversarial input triggers to analyze vulnerabilities in NLP models and their transferability across different models.

Title: Universal Adversarial Triggers for Attacking and Analyzing NLP

Abstract: Adversarial examples highlight model vulnerabilities and are useful for evaluation and interpretation. We define universal adversarial triggers: input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset. We propose a gradient-guided search over tokens which finds short trigger sequences (e.g., one word for classification and four words for language modeling) that successfully trigger the target prediction. For example, triggers cause SNLI entailment accuracy to drop from 89.94% to 0.55%, 72% of "why" questions in SQuAD to be answered "to kill american people", and the GPT-2 language model to spew racist output even when conditioned on non-racial contexts. Furthermore, although the triggers are optimized using white-box access to a specific model, they transfer to other models for all tasks we consider. Finally, since triggers are input-agnostic, they provide an analysis of global model behavior. For instance, they confirm that SNLI models exploit dataset biases and help to diagnose heuristics learned by reading comprehension models.