Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: Gauge fixing, BRS invariance and Ward identities for randomly stirred flows

Abstract: The Galilean invariance of the Navier-Stokes equation is shown to be akin to a global gauge symmetry familiar from quantum field theory. This symmetry leads to a multiple counting of infinitely many inertial reference frames in the path integral approach to randomly stirred fluids. This problem is solved by fixing the gauge, i.e., singling out one reference frame. The gauge fixed theory has an underlying Becchi-Rouet-Stora (BRS) symmetry which leads to the Ward identity relating the exact inverse response and vertex functions. This identification of Galilean invariance as a gauge symmetry is explored in detail, for different gauge choices and by performing a rigorous examination of a discretized version of the theory. The Navier-Stokes equation is also invariant under arbitrary rectilinear frame accelerations, known as extended Galilean invariance (EGI). We gauge fix this extended symmetry and derive the generalized Ward identity that follows from the BRS invariance of the gauge-fixed theory. This new Ward identity reduces to the standard one in the limit of zero acceleration. This gauge-fixing approach unambiguously shows that Galilean invariance and EGI constrain only the zero mode of the vertex but none of the higher wavenumber modes.


Prompt: Please find me the paper on training code large language models to follow instructions via git commits.

Output: Please find me the paper on training code large language models to follow instructions via git commits.

Output:
Title: Search for gravitational waves from binary black hole inspirals in LIGO data

Abstract: We report on a search for gravitational waves from binary black hole inspirals in the data from the second science run of the LIGO interferometers. The search focused on binary systems with component masses between 3 and 20 solar masses. Optimally oriented binaries with distances up to 1 Mpc could be detected with efficiency of at least 90%. We found no events that could be identified as gravitational waves in the 385.6 hours of data that we searched.


Prompt: Please find me the paper on training code large language models to follow instructions via git commits.

Output: Please find me the paper on training code large language models to follow instructions via git commits.

Output:
Title: Is the Commercial Mass Media Necessary, or Even Desirable, for Liberal Democracy?

Abstract: Is a commercial mass media, dependent on the market for its sustenance, necessary, or even desirable, for liberal democracy? Yochai Benkler has argued that a decentralized, peer-to-peer system of communications and information is both possible with digital technology and preferable to a system based on commercial mass media. He has contended in fact that the presence of politically powerful, copyright-rich mass media imposes significant barriers to the development of peer-to-peer information-sharing networks. In contrast, I have argued that the commercial mass media play an important, and perhaps even vital, role in liberal democracy by galvanizing public opinion, serving as a watchdog against government and corporate wrongdoing, agenda-setting (which enables public discourse), and serving as a relatively trustworthy source of information. This paper seeks to push the ball forward on this issue. It first isolates and enumerates the contributions that the commercial mass media are said to make towards liberal democracy. It then briefly assesses the extent to which the commercial mass media actually fulfills these constitutive functions. It then asks whether alternative institutions might serve some or all of these functions just as well or better. In so doing, it looks both to the past and the future. First, it examines the political party-supported press that thrived in the United States through much of the 19th century. Second, it examines government-funded mass media. Third, it looks, skeptically, at possibilities for peer-to-peer sharing of information and opinion in the digital network environment. I conclude that, despite the weaknesses of commercial mass media, an information policy suitable to liberal democracy should include a plurality of types of voices, including commercial mass media.


Prompt: Please find me the paper on training code large language models to follow instructions via git commits.

Output: Please find me the paper on training code large language models to follow instructions via git commits.

Output:
Title: Search for gravitational waves from binary black hole inspirals in LIGO data

Abstract: We report on a search for gravitational waves from binary black hole inspirals in the data from the second science run of the LIGO interferometers. The search focused on binary systems with component masses between 3 and 20 solar masses. Optimally oriented binaries with distances up to 1 Mpc could be detected with efficiency of at least 90%. We found no events that could be identified as gravitational waves in the 385.6 hours of data that we searched.


Prompt: Please find me the paper on training code large language models to follow instructions via git commits.

Output: Please find me the paper on training code large language models to follow instructions via git commits.

Output:
Title: InstructCoder: Instruction Tuning Large Language Models for Code Editing

Abstract: Code editing encompasses a variety of pragmatic tasks that developers deal with daily. Despite its relevance and practical usefulness, automatic code editing remains an underexplored area in the evolution of deep learning models, partly due to data scarcity. In this work, we explore the use of Large Language Models (LLMs) to edit code based on user instructions. Evaluated on a novel human-written execution-based benchmark dubbed EditEval, we found current models often struggle to fulfill the instructions. In light of this, we contribute InstructCoder, the first instruction-tuning dataset designed to adapt LLMs for general-purpose code editing, containing high-diversity code-editing tasks such as comment insertion, code optimization, and code refactoring. It consists of over 114,000 instruction-input-output triplets and covers multiple distinct code editing scenarios. The collection process starts with filtered commit data sourced from GitHub Python repositories as seeds. Subsequently, the dataset is systematically expanded through an iterative process, where both seed and generated tasks are used to prompt ChatGPT for more data. Our findings reveal that open-source LLMs fine-tuned on InstructCoder can significantly enhance the accuracy of code edits, exhibiting superior code-editing performance matching advanced proprietary LLMs. The datasets and the source code are publicly available at https://github.com/qishenghu/CodeInstruct.


Prompt: do you need a visa for a connecting flight in vietnam?

Output: do you need a visa for a connecting flight in vietnam?

Output:
Meta Stackexchange

Q: Stack Exchange site to ask Visa and passport related questions? Where can I ask questions about Passports and Visas?
Is there a Stack Exchange site for this?

A: travel's help centre says you can ask about visas.

customs & border crossing, visas and other travel documents such as permits

They also have a passports tag.
The Expatriates Stack Exchange is also available for people living or intend to live abroad on a long-term basis.
Do check out the help centre text of these sites before you ask to make sure your question is on topic.


Prompt: do you need a visa for a connecting flight in vietnam?

Output: do you need a visa for a connecting flight in vietnam?

Output:
Stackoverflow Stackexchange

Q: How to load data from splitted gzip files into redshift? Can I load data from splitted gzip files into an amazon redshift table?
I can load data from single gzip file or from splitted text files.
But can I load data from splitted gzip files?

A: I'm assuming here that you mean that you have multiple CSV files that are each gzipped.
First, upload each file to an S3 bucket under the same prefix and delimiter.
s3://S3_BUCKET/S3_PREFIX/file0.gz
s3://S3_BUCKET/S3_PREFIX/file1.gz
Then, execute the Redshift copy command:
copy TABLE_NAME from 's3://S3_BUCKET/S3_PREFIX' credentials 'aws_access_key_id=ACCESS_KEY;aws_secret_access_key=SECRET_KEY' csv gzip

*

*specify the S3 bucket and prefix

*include credentials that have permission to read the S3 objects

*ensure that the destination table already exists and has columns compatible with the CSV

*specify the csv and gzip options

Also, it's a good idea to have your file count proportional to the number of nodes in your cluster.


Prompt: How to reduce overoptimization in reinforcement learning from human feedback?

Output: How to reduce overoptimization in reinforcement learning from human feedback?

Output:
Title: Reward Model Ensembles Help Mitigate Overoptimization

Abstract: Reinforcement learning from human feedback (RLHF) is a standard approach for fine-tuning large language models to follow instructions. As part of this process, learned reward models are used to approximately model human preferences. However, as imperfect representations of the "true" reward, these learned reward models are susceptible to overoptimization. Gao et al. (2023) studied this phenomenon in a synthetic human feedback setup with a significantly larger "gold" reward model acting as the true reward (instead of humans) and showed that overoptimization remains a persistent problem regardless of the size of the proxy reward model and training data used. Using a similar setup, we conduct a systematic study to evaluate the efficacy of using ensemble-based conservative optimization objectives, specifically worst-case optimization (WCO) and uncertainty-weighted optimization (UWO), for mitigating reward model overoptimization when using two optimization methods: (a) best-of-n sampling (BoN) (b) proximal policy optimization (PPO). We additionally extend the setup of Gao et al. (2023) to include 25% label noise to better mirror real-world conditions. Both with and without label noise, we find that conservative optimization practically eliminates overoptimization and improves performance by up to 70% for BoN sampling. For PPO, ensemble-based conservative optimization always reduces overoptimization and outperforms single reward model optimization. Moreover, combining it with a small KL penalty successfully prevents overoptimization at no performance cost. Overall, our results demonstrate that ensemble-based conservative optimization can effectively counter overoptimization.


Prompt: How to reduce overoptimization in reinforcement learning from human feedback?

Output: How to reduce overoptimization in reinforcement learning from human feedback?

Output:
Title: Scalable Ensembling For Mitigating Reward Overoptimisation

Abstract: Reinforcement Learning from Human Feedback (RLHF) has enabled significant advancements within language modeling for powerful, instruction-following models. However, the alignment of these models remains a pressing challenge as the policy tends to overfit the learned ``proxy" reward model past an inflection point of utility as measured by a ``gold" reward model that is more performant -- a phenomenon known as overoptimisation. Prior work has mitigated this issue by computing a pessimistic statistic over an ensemble of reward models, which is common in Offline Reinforcement Learning but incredibly costly for language models with high memory requirements, making such approaches infeasible for sufficiently large models. To this end, we propose using a shared encoder but separate linear heads. We find this leads to similar performance as the full ensemble while allowing tremendous savings in memory and time required for training for models of similar size.


Prompt: what is the difference between domain users and authenticated users?

Output: what is the difference between domain users and authenticated users?

Output:
Stackoverflow Stackexchange

Q: Erlang: robustness against port owner's death What happens when the process which owns a spawned port dies and is restarted by the supervisor?

*

*Is there a way for the old port not to die with its owner and for the new owner to "take over"?

*Failing that, is it possible to ensure that the spawned process terminates when its port dies?

A: First, notice that you don't want the port owner to die. So move any "dangerous" code out to another process and make the port owner as dumb as possible. This is error-mitigation the Erlang way. Now, if that process dies, something is really bad, so in that case it may be clever to recycle the port as well. But since we moved everything out, we are counting on that not happening.
Regarding 2, the port will send a specific message when it terminates, so you can arrange for your spawned process to gracefully detect this and terminate with it. See
http://www.erlang.org/doc/reference_manual/ports.html


Prompt: what is the difference between domain users and authenticated users?

Output: what is the difference between domain users and authenticated users?

Output:
Stackoverflow Stackexchange

Q: What are the differences between LDAP and Active Directory authentication? I have tried to search without success.
(I do know difference between LDAP and Active Directory. Also read this What are the differences between LDAP and Active Directory?)
According to my understanding, LDAP authenticates (binds) with DN (distinguish name) and password.
E.g. CN=bob, OU=Users,DC=test,DC=com
Active Directory authenticates with a short user name or full domain user name and password.
E.g. bob or bob@test.com
Therefore, LDAP does not allow to users with the same CN in the same tree (OU) but it allows the same CN in a different OU.
Active Directory does not allow the same user name in the same domain, even the user object located in different tree of Active Directory.
Active Directory allows the same user name in different domains.
Other differences?

A: Active Directory is a product and a directory service. LDAP is an application layer protocol and an IETF standard. Active Directory supports and implements LDAP, therefore allowing RFC-compliant LDAP clients to talk to it and perform LDAP operations such as bind, search, etc.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: Gauge fixing, BRS invariance and Ward identities for randomly stirred flows

Abstract: The Galilean invariance of the Navier-Stokes equation is shown to be akin to a global gauge symmetry familiar from quantum field theory. This symmetry leads to a multiple counting of infinitely many inertial reference frames in the path integral approach to randomly stirred fluids. This problem is solved by fixing the gauge, i.e., singling out one reference frame. The gauge fixed theory has an underlying Becchi-Rouet-Stora (BRS) symmetry which leads to the Ward identity relating the exact inverse response and vertex functions. This identification of Galilean invariance as a gauge symmetry is explored in detail, for different gauge choices and by performing a rigorous examination of a discretized version of the theory. The Navier-Stokes equation is also invariant under arbitrary rectilinear frame accelerations, known as extended Galilean invariance (EGI). We gauge fix this extended symmetry and derive the generalized Ward identity that follows from the BRS invariance of the gauge-fixed theory. This new Ward identity reduces to the standard one in the limit of zero acceleration. This gauge-fixing approach unambiguously shows that Galilean invariance and EGI constrain only the zero mode of the vertex but none of the higher wavenumber modes.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: Text and Code Embeddings by Contrastive Pre-Training

Abstract: Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture. In this work, we show that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code. The same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine-tuned models. On linear-probe classification accuracy averaging over 7 tasks, our best unsupervised model achieves a relative improvement of 4% and 1.8% over previous best unsupervised and supervised text embedding models respectively. The same text embeddings when evaluated on large-scale semantic search attains a relative improvement of 23.4%, 14.7%, and 10.6% over previous best unsupervised methods on MSMARCO, Natural Questions and TriviaQA benchmarks, respectively. Similarly to text embeddings, we train code embedding models on (text, code) pairs, obtaining a 20.8% relative improvement over prior best work on code search.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: Gauge fixing, BRS invariance and Ward identities for randomly stirred flows

Abstract: The Galilean invariance of the Navier-Stokes equation is shown to be akin to a global gauge symmetry familiar from quantum field theory. This symmetry leads to a multiple counting of infinitely many inertial reference frames in the path integral approach to randomly stirred fluids. This problem is solved by fixing the gauge, i.e., singling out one reference frame. The gauge fixed theory has an underlying Becchi-Rouet-Stora (BRS) symmetry which leads to the Ward identity relating the exact inverse response and vertex functions. This identification of Galilean invariance as a gauge symmetry is explored in detail, for different gauge choices and by performing a rigorous examination of a discretized version of the theory. The Navier-Stokes equation is also invariant under arbitrary rectilinear frame accelerations, known as extended Galilean invariance (EGI). We gauge fix this extended symmetry and derive the generalized Ward identity that follows from the BRS invariance of the gauge-fixed theory. This new Ward identity reduces to the standard one in the limit of zero acceleration. This gauge-fixing approach unambiguously shows that Galilean invariance and EGI constrain only the zero mode of the vertex but none of the higher wavenumber modes.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Prompt: I am looking for a paper to help me deal with data constraints as I scale up my language model.

Output: I am looking for a paper to help me deal with data constraints as I scale up my language model.

Output:
Title: Scaling Data-Constrained Language Models

Abstract: The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Prompt: Please find me the paper on training code large language models to follow instructions via git commits.

Output: Please find me the paper on training code large language models to follow instructions via git commits.

Output:
Title: OctoPack: Instruction Tuning Code Large Language Models

Abstract: Finetuning large language models (LLMs) on instructions leads to vast performance improvements on natural language tasks. We apply instruction tuning using code, leveraging the natural structure of Git commits, which pair code changes with human instructions. We compile CommitPack: 4 terabytes of Git commits across 350 programming languages. We benchmark CommitPack against other natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B parameter StarCoder model, and achieve state-of-the-art performance among models not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2% pass@1). We further introduce HumanEvalPack, expanding the HumanEval benchmark to a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis) across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Our models, OctoCoder and OctoGeeX, achieve the best performance across HumanEvalPack among all permissive models, demonstrating CommitPack's benefits in generalizing to a wider set of languages and natural coding tasks. Code, models and data are freely available at https://github.com/bigcode-project/octopack.


Prompt: do crops respawn in skyrim?

Output: do crops respawn in skyrim?

Output:
Meta Stackexchange

Q: Google time and answer mistakes Take a look a this :

On the google search page I can see

*

*13 hours ago

*1 answer

When I actually posted the question :

*

*8 mins ago

*0 answers

This should be fixed.


Prompt: do crops respawn in skyrim?

Output: do crops respawn in skyrim?

Output:
Gaming Stackexchange

Q: Do Nirnroots regrow in Skyrim? In Oblivion, Nirnroots were a collectible - there was a finite number of them*, and they did not respawn.
I've been noticing, however, that when I pick Nirnroots in Skyrim, the roots of the plant remain behind. This got me wondering...
Do Nirnroots regrow in Skyrim?
What about the Crimson Nirnroots of Blackreach?
*discounting the respawning food-nirnroot in Frostcrag Spire

A: I can confirm they grow back after 15 days also. I'm using the PS3 version and picked about 20 of them before doing the related quest. It does give you a negative number if you try dropping them and picking them back up. So, I was forced to wait 15 days to pick a total of 50 of them.


Prompt: why star trek tng ended?

Output: why star trek tng ended?

Output:
Scifi Stackexchange

Q: Why did Star Trek: The Next Generation season 2 only have 22 episodes? In the Star Trek franchise shows Star Trek: The Next Generation, Star Trek: Deep Space Nine and Star Trek: Voyager, each show almost always had 26 episodes per season.  Besides Voyager's first season, the only exception in this is the second season of Star Trek: The Next Generation, which instead had 22.  Why is this?

A: There was a writers' strike in 1988.
The Writers´ Guild of America went out on strike in the spring and summer of 1988. They demanded more creative control and residuals for the shows they wrote.
The writers of ST:TNG at Paramount joined in despite their Sacred Duty.
The strike ended on August 7, but the production pipeline was adversely affected, and ST:TNG wasn't able to start airing until November 21. (This episode, "The Child", contained the first appearance of Whoopi Goldberg playing Guinan.)


Prompt: why star trek tng ended?

Output: why star trek tng ended?

Output:
Mathematica Stackexchange

Q: how to download Finite ELements tutorial as PDF For long tutorials, I prefer hard copy.
Is it possible to download from somewhere this document
https://reference.wolfram.com/language/FEMDocumentation/tutorial/FiniteElementOverview.html
as PDF file?  Few years ago WRI used to have PDF files for all its documentations, but I am not able to find one for the above now.


Prompt: Paper on SEACrowd

Output: Paper on SEACrowd

Output:
Title: SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages

Abstract: Southeast Asia (SEA) is a region rich in linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA, compromising the quality of AI models for SEA languages. Evaluating models for SEA languages is challenging due to the scarcity of high-quality datasets, compounded by the dominance of English training data, raising concerns about potential cultural misrepresentation. To address these challenges, we introduce SEACrowd, a collaborative initiative that consolidates a comprehensive resource hub that fills the resource gap by providing standardized corpora in nearly 1,000 SEA languages across three modalities. Through our SEACrowd benchmarks, we assess the quality of AI models on 36 indigenous languages across 13 tasks, offering valuable insights into the current AI landscape in SEA. Furthermore, we propose strategies to facilitate greater AI advancements, maximizing potential utility and resource equity for the future of AI in SEA.


Prompt: Paper on SEACrowd

Output: Paper on SEACrowd

Output:
Title: Carriers of \emph{Sargassum} and mechanism for coastal inundation in the Caribbean Sea

Abstract: We identify effective carriers of \emph{Sargassum} in the Caribbean Sea and describe a mechanism for coastal choking. Revealed from satellite altimetry, the carriers of \emph{Sargassum} are mesoscale eddies (vortices of 50-km radius or larger) with coherent material (i.e., fluid) boundaries. These are observer-independent -- \emph{unlike} eddy boundaries identified with instantaneously closed streamlines of the altimetric sea-surface height field -- and furthermore harbor finite-time attractors for networks of elastically connected finite-size buoyant or "inertial" particles dragged by ocean currents and winds, a mathematical abstraction of \emph{Sargassum} rafts. The mechanism of coastal inundation, identified using a minimal model of surface-intensified Caribbean Sea eddies, is thermal instability in the presence of bottom topography.


Prompt: Paper that shows multitask finetuned language models can generalize to unseen languages

Output: Paper that shows multitask finetuned language models can generalize to unseen languages

Output:
Title: Constructive Complexity and Artificial Reality: An Introduction

Abstract: Basic problems of complex systems are outlined with an emphasis on irreducibility and dynamic many-to-many correspondences. We discuss the importance of a constructive approach to artificial reality and the significance of an internal observer.


Prompt: Paper that shows multitask finetuned language models can generalize to unseen languages

Output: Paper that shows multitask finetuned language models can generalize to unseen languages

Output:
Title: The Absence of Positive Energy Bound States for a Class of Nonlocal Potentials

Abstract: We generalize in this paper a theorem of Titchmarsh for the positivity of Fourier sine integrals. We apply then the theorem to derive simple conditions for the absence of positive energy bound states (bound states embedded in the continuum) for the radial Schr\"odinger equation with nonlocal potentials which are superposition of a local potential and separable potentials.


Prompt: Paper that introduced GPT-3

Output: Paper that introduced GPT-3

Output:
Title: a survey on GPT-3

Abstract: This paper provides an introductory survey to GPT-3. We cover some of the historical development behind this technology, some of the key features of GPT-3, and discuss the machine learning model and the datasets used. We survey both academic and commercial efforts applying GPT-3 in diverse domains such as developing conversational AI chatbots, software development, creative work, domain knowledge, and business productivity. We discuss some of the challenges that GPT-3 faces such as the problems of training complexity, bias, and hallucination/incorrect answers. We also discuss the future research opportunities in this area.


Prompt: Paper that introduced GPT-3

Output: Paper that introduced GPT-3

Output:
Title: Improving Short Text Classification With Augmented Data Using GPT-3

Abstract: GPT-3 is a large-scale natural language model developed by OpenAI that can perform many different tasks, including topic classification. Although researchers claim that it requires only a small number of in-context examples to learn a task, in practice GPT-3 requires these training examples to be either of exceptional quality or a higher quantity than easily created by hand. To address this issue, this study teaches GPT-3 to classify whether a question is related to data science by augmenting a small training set with additional examples generated by GPT-3 itself. This study compares two classifiers: the GPT-3 Classification Endpoint with augmented examples, and the GPT-3 Completion Endpoint with an optimal training set chosen using a genetic algorithm. We find that while the augmented Completion Endpoint achieves upwards of 80 percent validation accuracy, using the augmented Classification Endpoint yields more consistent accuracy on unseen examples. In this way, giving large-scale machine learning models like GPT-3 the ability to propose their own additional training examples can result in improved classification performance.


Prompt: paper showing crosslingual generalization is possible

Output: paper showing crosslingual generalization is possible

Output:
Title: Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization

Abstract: Large language models (LLMs) have exhibited considerable cross-lingual generalization abilities, whereby they implicitly transfer knowledge across languages. However, the transfer is not equally successful for all languages, especially for low-resource ones, which poses an ongoing challenge. It is unclear whether we have reached the limits of implicit cross-lingual generalization and if explicit knowledge transfer is viable. In this paper, we investigate the potential for explicitly aligning conceptual correspondence between languages to enhance cross-lingual generalization. Using the syntactic aspect of language as a testbed, our analyses of 43 languages reveal a high degree of alignability among the spaces of structural concepts within each language for both encoder-only and decoder-only LLMs. We then propose a meta-learning-based method to learn to align conceptual spaces of different languages, which facilitates zero-shot and few-shot generalization in concept classification and also offers insights into the cross-lingual in-context learning phenomenon. Experiments on syntactic analysis tasks show that our approach achieves competitive results with state-of-the-art methods and narrows the performance gap between languages, particularly benefiting those with limited resources.


Prompt: paper showing crosslingual generalization is possible

Output: paper showing crosslingual generalization is possible

Output:
Title: Asymptotic Exit Location Distributions in the Stochastic Exit Problem

Abstract: Consider a two-dimensional continuous-time dynamical system, with an attracting fixed point $S$. If the deterministic dynamics are perturbed by white noise (random perturbations) of strength $\epsilon$, the system state will eventually leave the domain of attraction $\Omega$ of $S$. We analyse the case when, as $\epsilon\to0$, the exit location on the boundary $\partial\Omega$ is increasingly concentrated near a saddle point $H$ of the deterministic dynamics. We show that the asymptotic form of the exit location distribution on $\partial\Omega$ is generically non-Gaussian and asymmetric, and classify the possible limiting distributions. A key role is played by a parameter $\mu$, equal to the ratio $|\lambda_s(H)|/\lambda_u(H)$ of the stable and unstable eigenvalues of the linearized deterministic flow at $H$. If $\mu<1$ then the exit location distribution is generically asymptotic as $\epsilon\to0$ to a Weibull distribution with shape parameter $2/\mu$, on the $O(\epsilon^{\mu/2})$ length scale near $H$. If $\mu>1$ it is generically asymptotic to a distribution on the $O(\epsilon^{1/2})$ length scale, whose moments we compute. The asymmetry of the asymptotic exit location distribution is attributable to the generic presence of a `classically forbidden' region: a wedge-shaped subset of $\Omega$ with $H$ as vertex, which is reached from $S$, in the $\epsilon\to0$ limit, only via `bent' (non-smooth) fluctuational paths that first pass through the vicinity of $H$. We deduce from the presence of this forbidden region that the classical Eyring formula for the small-$\epsilon$ exponential asymptotics of the mean first exit time is generically inapplicable.


Prompt: are all naruto shippuden episodes dubbed yet?

Output: are all naruto shippuden episodes dubbed yet?

Output:
Scifi Stackexchange

Q: What is the only Jutsu included in the manga from Filler Episodes of Naruto Shippuden Anime My friend challenged me with this question a while ago.
Which jutsu premiered in Naruto Shippuden, but was thereafter picked up by Masashi Kishimoto into his Manga?
This has apparently only happened once, all the other jutsu's happened the other way.  He created them and put them into the Manga, then were disseminated into the Shippuden

A: The only jutsu that matches somehow the requirements (at least to my knowledge and search) is the Water Release: Water Shuriken (Suiton: Mizu Shuriken):

This jutsu appeared first in Naruto: Shippūden #412 - Neji's Judgment which aired in Japan on May 14, 2015 and was later also used in the Naruto novel Gaara Hiden: A Sandstorm Mirage which was published in Japan on June 4, 2015.
However this is a novel and not the original manga although is was illustrated by Masashi Kishimoto.


Prompt: are all naruto shippuden episodes dubbed yet?

Output: are all naruto shippuden episodes dubbed yet?

Output:
Scifi Stackexchange

Q: What is the only Jutsu included in the manga from Filler Episodes of Naruto Shippuden Anime My friend challenged me with this question a while ago.
Which jutsu premiered in Naruto Shippuden, but was thereafter picked up by Masashi Kishimoto into his Manga?
This has apparently only happened once, all the other jutsu's happened the other way.  He created them and put them into the Manga, then were disseminated into the Shippuden

A: The only jutsu that matches somehow the requirements (at least to my knowledge and search) is the Water Release: Water Shuriken (Suiton: Mizu Shuriken):

This jutsu appeared first in Naruto: Shippūden #412 - Neji's Judgment which aired in Japan on May 14, 2015 and was later also used in the Naruto novel Gaara Hiden: A Sandstorm Mirage which was published in Japan on June 4, 2015.
However this is a novel and not the original manga although is was illustrated by Masashi Kishimoto.


Prompt: what is the difference between phd and pdf?

Output: what is the difference between phd and pdf?

Output:
Stackoverflow Stackexchange

Q: Importing Drupal 5 content into Drupal 6 I need to import several thousand nodes from a Drupal 5 sql dump into a Drupal 6 website. I tried using Node Import and Import/Export API but very little success... any ideas as to how this can be done?
Many thanks!
Jane

A: I used Views Bonus Pack on a Drupal 5 site to export the nodes into a csv file.
Then I used the feeds module to import the csv into a Drupal 6 site.


Prompt: what is the difference between phd and pdf?

Output: what is the difference between phd and pdf?

Output:
Math Stackexchange

Q: Difference between a PDF and PMF What is the difference between a probability density function and a probability mass function?

A: One is for discrete variables, the other for continuous.
They are also interpreted differently. The PDF is a probability density. If f(x) is a PDF, f(x) doesn't tell you the probability of getting x (In fact, the probability of getting precisely x is 0).
The way you use a PDF is through integration. So, the probability of getting a value that is in a dx area around x is approximately f(x)*dx.


Prompt: Unified-IO model

Output: Unified-IO model

Output:
Title: Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

Abstract: We present Unified-IO 2, the first autoregressive multimodal model that is capable of understanding and generating image, text, audio, and action. To unify different modalities, we tokenize inputs and outputs -- images, text, audio, action, bounding boxes, etc., into a shared semantic space and then process them with a single encoder-decoder transformer model. Since training with such diverse modalities is challenging, we propose various architectural improvements to stabilize model training. We train our model from scratch on a large multimodal pre-training corpus from diverse sources with a multimodal mixture of denoisers objective. To learn an expansive set of skills, such as following multimodal instructions, we construct and finetune on an ensemble of 120 datasets with prompts and augmentations. With a single unified model, Unified-IO 2 achieves state-of-the-art performance on the GRIT benchmark and strong results in more than 35 benchmarks, including image generation and understanding, natural language understanding, video and audio understanding, and robotic manipulation. We release all our models to the research community.


Prompt: Unified-IO model

Output: Unified-IO model

Output:
Title: A Unified Programming Model for Heterogeneous Computing with CPU and Accelerator Technologies

Abstract: This paper consists of three parts. The first part provides a unified programming model for heterogeneous computing with CPU and accelerator (like GPU, FPGA, Google TPU, Atos QPU, and more) technologies. To some extent, this new programming model makes programming across CPUs and accelerators turn into usual programming tasks with common programming languages, and relieves complexity of programming across CPUs and accelerators. It can be achieved by extending file managements in common programming languages, such as C/C++, Fortran, Python, MPI, etc., to cover accelerators as I/O devices. In the second part, we show that all types of computer systems can be reduced to the simplest type of computer system, a single-core CPU computer system with I/O devices, by the unified programming model. Thereby, the unified programming model can truly build the programming of various computer systems on one API (i.e. file managements of common programming languages), and can make programming for various computer systems easier. In third part, we present a new approach to coupled applications computing (like multidisciplinary simulations) by the unified programming model. The unified programming model makes coupled applications computing more natural and easier since it only relies on its own power to couple multiple applications through MPI.


Prompt: Good benchmark for multitask performance of llms

Output: Good benchmark for multitask performance of llms

Output:
Title: Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?

Abstract: Large language models (LLMs) are typically prompted to follow a single instruction per inference call. In this work, we analyze whether LLMs also hold the capability to handle multiple instructions simultaneously, denoted as Multi-Task Inference. For this purpose, we introduce the MTI Bench(Multi-Task Inference Benchmark), a comprehensive evaluation benchmark encompassing 5,000 instances across 25 tasks. Each task in the MTI Bench involves 2 to 3 sub-tasks. As expected, we first demonstrate that Multi-Task Inference reduces the total inference time by 1.46 times in average since it does not require multiple inference calls. Interestingly, contrary to the expectation that LLMs would perform better when tasks are divided, we find that state-of-the-art LLMs, such as Llama-2-Chat-70B and GPT-4, show up to 7.3% and 12.4% improved performance with Multi-Task Inference compared to Single-Task Inference on the MTI Bench. We release the MTI Bench dataset and our code at this link https://github.com/guijinSON/MTI-Bench.


Prompt: Good benchmark for multitask performance of llms

Output: Good benchmark for multitask performance of llms

Output:
Title: Limit theorems and ergodicity for general bootstrap random walks

Abstract: Given the increments of a simple symmetric random walk $(X_n)_{n\ge0}$, we characterize all possible ways of recycling these increments into a simple symmetric random walk $(Y_n)_{n\ge0}$ adapted to the filtration of $(X_n)_{n\ge0}$. We study the long term behavior of a suitably normalized two-dimensional process $((X_n,Y_n))_{n\ge0}$. In particular, we provide necessary and sufficient conditions for the process to converge to a two-dimensional Brownian motion (possibly degenerate). We also discuss cases in which the limit is not Gaussian. Finally, we provide a simple necessary and sufficient condition for the ergodicity of the recycling transformation, thus generalizing results from Dubins and Smorodinsky (1992) and Fujita (2008), and solving the discrete version of the open problem of the ergodicity of the general L\'evy transformation (see Mansuy and Yor, 2006).


Prompt: A paper evaluating whether updating one fact in a language model affects related facts.

Output: A paper evaluating whether updating one fact in a language model affects related facts.

Output:
Title: Evaluating the Ripple Effects of Knowledge Editing in Language Models

Abstract: Modern language models capture a large body of factual knowledge. However, some facts can be incorrectly induced or become obsolete over time, resulting in factually incorrect generations. This has led to the development of various editing methods that allow updating facts encoded by the model. Evaluation of these methods has primarily focused on testing whether an individual fact has been successfully injected, and if similar predictions for other subjects have not changed. Here we argue that such evaluation is limited, since injecting one fact (e.g. ``Jack Depp is the son of Johnny Depp'') introduces a ``ripple effect'' in the form of additional facts that the model needs to update (e.g.``Jack Depp is the sibling of Lily-Rose Depp''). To address this issue, we propose a novel set of evaluation criteria that consider the implications of an edit on related facts. Using these criteria, we then construct RippleEdits, a diagnostic benchmark of 5K factual edits, capturing a variety of types of ripple effects. We evaluate prominent editing methods on RippleEdits, showing that current methods fail to introduce consistent changes in the model's knowledge. In addition, we find that a simple in-context editing baseline obtains the best scores on our benchmark, suggesting a promising research direction for model editing.


Prompt: A paper evaluating whether updating one fact in a language model affects related facts.

Output: A paper evaluating whether updating one fact in a language model affects related facts.

Output:
Title: WikiFactDiff: A Large, Realistic, and Temporally Adaptable Dataset for Atomic Factual Knowledge Update in Causal Language Models

Abstract: The factuality of large language model (LLMs) tends to decay over time since events posterior to their training are "unknown" to them. One way to keep models up-to-date could be factual update: the task of inserting, replacing, or removing certain simple (atomic) facts within the model. To study this task, we present WikiFactDiff, a dataset that describes the evolution of factual knowledge between two dates as a collection of simple facts divided into three categories: new, obsolete, and static. We describe several update scenarios arising from various combinations of these three types of basic update. The facts are represented by subject-relation-object triples; indeed, WikiFactDiff was constructed by comparing the state of the Wikidata knowledge base at 4 January 2021 and 27 February 2023. Those fact are accompanied by verbalization templates and cloze tests that enable running update algorithms and their evaluation metrics. Contrary to other datasets, such as zsRE and CounterFact, WikiFactDiff constitutes a realistic update setting that involves various update scenarios, including replacements, archival, and new entity insertions. We also present an evaluation of existing update algorithms on WikiFactDiff.


Prompt: I'm looking for a QA model that retrieves both textual and visual information.

Output: I'm looking for a QA model that retrieves both textual and visual information.

Output:
Title: Look Before you Speak: Visually Contextualized Utterances

Abstract: While most conversational AI systems focus on textual dialogue only, conditioning utterances on visual context (when it's available) can lead to more realistic conversations. Unfortunately, a major challenge for incorporating visual context into conversational dialogue is the lack of large-scale labeled datasets. We provide a solution in the form of a new visually conditioned Future Utterance Prediction task. Our task involves predicting the next utterance in a video, using both visual frames and transcribed speech as context. By exploiting the large number of instructional videos online, we train a model to solve this task at scale, without the need for manual annotations. Leveraging recent advances in multimodal learning, our model consists of a novel co-attentional multimodal video transformer, and when trained on both textual and visual context, outperforms baselines that use textual inputs alone. Further, we demonstrate that our model trained for this task on unlabelled videos achieves state-of-the-art performance on a number of downstream VideoQA benchmarks such as MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA.


Prompt: I'm looking for a QA model that retrieves both textual and visual information.

Output: I'm looking for a QA model that retrieves both textual and visual information.

Output:
Title: Question Aware Vision Transformer for Multimodal Reasoning

Abstract: Vision-Language (VL) models have gained significant research focus, enabling remarkable advances in multimodal reasoning. These architectures typically comprise a vision encoder, a Large Language Model (LLM), and a projection module that aligns visual features with the LLM's representation space. Despite their success, a critical limitation persists: the vision encoding process remains decoupled from user queries, often in the form of image-related questions. Consequently, the resulting visual features may not be optimally attuned to the query-specific elements of the image. To address this, we introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal reasoning, which embeds question awareness directly within the vision encoder. This integration results in dynamic visual features focusing on relevant image aspects to the posed question. QA-ViT is model-agnostic and can be incorporated efficiently into any VL architecture. Extensive experiments demonstrate the effectiveness of applying our method to various multimodal architectures, leading to consistent improvement across diverse tasks and showcasing its potential for enhancing visual and scene-text understanding.


Prompt: A method using descriptors from large language models to improve accuracy and mitigate bias in visual classification tasks.

Output: A method using descriptors from large language models to improve accuracy and mitigate bias in visual classification tasks.

Output:
Title: Images in Language Space: Exploring the Suitability of Large Language Models for Vision & Language Tasks

Abstract: Large language models have demonstrated robust performance on various language tasks using zero-shot or few-shot learning paradigms. While being actively researched, multimodal models that can additionally handle images as input have yet to catch up in size and generality with language-only models. In this work, we ask whether language-only models can be utilised for tasks that require visual input -- but also, as we argue, often require a strong reasoning component. Similar to some recent related work, we make visual information accessible to the language model using separate verbalisation models. Specifically, we investigate the performance of open-source, open-access language models against GPT-3 on five vision-language tasks when given textually-encoded visual information. Our results suggest that language models are effective for solving vision-language tasks even with limited samples. This approach also enhances the interpretability of a model's output by providing a means of tracing the output back through the verbalised image content.


Prompt: A method using descriptors from large language models to improve accuracy and mitigate bias in visual classification tasks.

Output: A method using descriptors from large language models to improve accuracy and mitigate bias in visual classification tasks.

Output:
Title: Visual Classification via Description from Large Language Models

Abstract: Vision-language models (VLMs) such as CLIP have shown promising performance on a variety of recognition tasks using the standard zero-shot classification procedure -- computing similarity between the query image and the embedded words for each category. By only using the category name, they neglect to make use of the rich context of additional information that language affords. The procedure gives no intermediate understanding of why a category is chosen, and furthermore provides no mechanism for adjusting the criteria used towards this decision. We present an alternative framework for classification with VLMs, which we call classification by description. We ask VLMs to check for descriptive features rather than broad categories: to find a tiger, look for its stripes; its claws; and more. By basing decisions on these descriptors, we can provide additional cues that encourage using the features we want to be used. In the process, we can get a clear idea of what features the model uses to construct its decision; it gains some level of inherent explainability. We query large language models (e.g., GPT-3) for these descriptors to obtain them in a scalable way. Extensive experiments show our framework has numerous advantages past interpretability. We show improvements in accuracy on ImageNet across distribution shifts; demonstrate the ability to adapt VLMs to recognize concepts unseen during training; and illustrate how descriptors can be edited to effectively mitigate bias compared to the baseline.


Prompt: Highest BLEU scores for fully NAT in WMT14 En-De with faster inference.

Output: Highest BLEU scores for fully NAT in WMT14 En-De with faster inference.

Output:
Title: Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade

Abstract: Fully non-autoregressive neural machine translation (NAT) is proposed to simultaneously predict tokens with single forward of neural networks, which significantly reduces the inference latency at the expense of quality drop compared to the Transformer baseline. In this work, we target on closing the performance gap while maintaining the latency advantage. We first inspect the fundamental issues of fully NAT models, and adopt dependency reduction in the learning space of output tokens as the basic guidance. Then, we revisit methods in four different aspects that have been proven effective for improving NAT models, and carefully combine these techniques with necessary modifications. Our extensive experiments on three translation benchmarks show that the proposed system achieves the new state-of-the-art results for fully NAT models, and obtains comparable performance with the autoregressive and iterative NAT systems. For instance, one of the proposed models achieves 27.49 BLEU points on WMT14 En-De with approximately 16.5X speed up at inference time.


Prompt: Highest BLEU scores for fully NAT in WMT14 En-De with faster inference.

Output: Highest BLEU scores for fully NAT in WMT14 En-De with faster inference.

Output:
Title: Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade

Abstract: Fully non-autoregressive neural machine translation (NAT) is proposed to simultaneously predict tokens with single forward of neural networks, which significantly reduces the inference latency at the expense of quality drop compared to the Transformer baseline. In this work, we target on closing the performance gap while maintaining the latency advantage. We first inspect the fundamental issues of fully NAT models, and adopt dependency reduction in the learning space of output tokens as the basic guidance. Then, we revisit methods in four different aspects that have been proven effective for improving NAT models, and carefully combine these techniques with necessary modifications. Our extensive experiments on three translation benchmarks show that the proposed system achieves the new state-of-the-art results for fully NAT models, and obtains comparable performance with the autoregressive and iterative NAT systems. For instance, one of the proposed models achieves 27.49 BLEU points on WMT14 En-De with approximately 16.5X speed up at inference time.


Prompt: A comprehensive analysis of a 280 billion parameter language model and its performance across 152 diverse tasks

Output: A comprehensive analysis of a 280 billion parameter language model and its performance across 152 diverse tasks

Output:
Title: Baichuan 2: Open Large-scale Language Models

Abstract: Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering. However, most powerful LLMs are closed-source or limited in their capability for languages other than English. In this technical report, we present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan 2 excels in vertical domains such as medicine and law. We will release all pre-training model checkpoints to benefit the research community in better understanding the training dynamics of Baichuan 2.


Prompt: A comprehensive analysis of a 280 billion parameter language model and its performance across 152 diverse tasks

Output: A comprehensive analysis of a 280 billion parameter language model and its performance across 152 diverse tasks

Output:
Title: Seq2Mol: Automatic design of de novo molecules conditioned by the target protein sequences through deep neural networks

Abstract: De novo design of molecules has recently enjoyed the power of generative deep neural networks. Current approaches aim to generate molecules either resembling the properties of the molecules of the training set or molecules that are optimized with respect to specific physicochemical properties. None of the methods generates molecules specific to a target protein. In the approach presented here, we introduce a method which is conditioned on the protein target sequence to generate de novo molecules that are relevant to the target. We use an implementation adapted from Google's "Show and Tell" image caption generation method, to generate SMILES strings of molecules from protein sequence embeddings generated by a deep bi-directional language model ELMo. ELMo is used to generate contextualized embedding vectors of the protein sequence. Using reinforcement learning, the trained model is further optimized through augmented episodic likelihood to increase the diversity of the generated compounds compared to the training set. We used the model to generate compounds for two major drug target families, i.e. for GPCRs and Tyrosine Kinase targets. The model generated compounds which are structurally different form the training set, while also being more similar to compounds known to bind to the two families of drug targets compared to a random set of molecules. The compounds further display reasonable synthesizability and drug-likeness scores.


Prompt: Paper introducing a model connecting object segmentation masks with natural language responses in multimodal settings.

Output: Paper introducing a model connecting object segmentation masks with natural language responses in multimodal settings.

Output:
Title: Dynamic Multimodal Instance Segmentation guided by natural language queries

Abstract: We address the problem of segmenting an object given a natural language expression that describes it. Current techniques tackle this task by either (\textit{i}) directly or recursively merging linguistic and visual information in the channel dimension and then performing convolutions; or by (\textit{ii}) mapping the expression to a space in which it can be thought of as a filter, whose response is directly related to the presence of the object at a given spatial coordinate in the image, so that a convolution can be applied to look for the object. We propose a novel method that integrates these two insights in order to fully exploit the recursive nature of language. Additionally, during the upsampling process, we take advantage of the intermediate information generated when downsampling the image, so that detailed segmentations can be obtained. We compare our method against the state-of-the-art approaches in four standard datasets, in which it surpasses all previous methods in six of eight of the splits for this task.


Prompt: Paper introducing a model connecting object segmentation masks with natural language responses in multimodal settings.

Output: Paper introducing a model connecting object segmentation masks with natural language responses in multimodal settings.

Output:
Title: Comprehensive Multi-Modal Interactions for Referring Image Segmentation

Abstract: We investigate Referring Image Segmentation (RIS), which outputs a segmentation map corresponding to the natural language description. Addressing RIS efficiently requires considering the interactions happening across visual and linguistic modalities and the interactions within each modality. Existing methods are limited because they either compute different forms of interactions sequentially (leading to error propagation) or ignore intramodal interactions. We address this limitation by performing all three interactions simultaneously through a Synchronous Multi-Modal Fusion Module (SFM). Moreover, to produce refined segmentation masks, we propose a novel Hierarchical Cross-Modal Aggregation Module (HCAM), where linguistic features facilitate the exchange of contextual information across the visual hierarchy. We present thorough ablation studies and validate our approach's performance on four benchmark datasets, showing considerable performance gains over the existing state-of-the-art (SOTA) methods.


Prompt: age is what type of scale?

Output: age is what type of scale?

Output:
Stats Stackexchange

Q: Should I treat age as a continuous variable or factor? Age is a predictor variable among a few others (gender, weight and height) and my response is the log-odds of a certain disease (binomial glm).
My age data runs from 21 until 40. I am not sure whether to treat age as a continuous variable or as a factor with age groups: 21-25, 26-30, 31-35, 36-40.
Are there any plots which I can produce that can help determine which would be the better approach?

A: It depends on the context.
For example if you are looking for the effect of age on children's height, it makes sense to look at it as a continuous ( integer) value.
If you're looking for e.g. the effect of age on oncogenesis then it makes sense if you look at age groups. Young vs old, above 55 and below 55, ...
For your example, unless age is a confounder of a hidden factor such as for example being college grad or still a student ( risk factor for young adults STD infection), I'd bin my data into reasonable bin sizes.


Prompt: age is what type of scale?

Output: age is what type of scale?

Output:
Stats Stackexchange

Q: Are the variable types here considered correct? If we want to determine the variable types, will it be as follows for the below variables?
Age ---> quantitative, discrete (we can count)
Fitness ---> If the values that we will enter here are 0 and 1 only, will the type of this variable be qualitative, nominal?
Thanks a lot.

A: Questions like this expose the problems with Stevens' typology of variables. Some variables just aren't any of his categories; or are hard to categorize. Age is ratio level data in the sense that someone who is (say) 40 years old is twice as old as someone who is 20 years old. You just haven't got it measured precisely. But so? What difference does that make?
Dichotomies - here it really doesn't matter HOW you treat them for most purposes. E.g. in regression, you can consider them to be nominal, ordinal or interval and you will get the same results.
I wrote about other problems with this typology on my blog


Prompt: A framework to improve logical problem-solving in language models integrating symbolic formulation and self-refinement.

Output: A framework to improve logical problem-solving in language models integrating symbolic formulation and self-refinement.

Output:
Title: On the three state Potts model with competing interactions on the Bethe lattice

Abstract: In the present paper the three state Potts model with competing binary interactions (with couplings $J$ and $J_p$) on the second order Bethe lattice is considered. The recurrent equations for the partition functions are derived. When $J_p=0$, by means of a construction of a special class of limiting Gibbs measures, it is shown how these equations are related with the surface energy of the Hamiltonian. This relation reduces the problem of describing the limit Gibbs measures to find of solutions of a nonlinear functional equation. Moreover, the set of ground states of the one-level model is completely described. Using this fact, one finds Gibbs measures (pure phases) associated with the translation-invariant ground states. The critical temperature is exactly found and the phase diagram is presented. The free energies corresponding to translations-invariant Gibbs measures are found. Certain physical quantities are calculated as well.


Prompt: A framework to improve logical problem-solving in language models integrating symbolic formulation and self-refinement.

Output: A framework to improve logical problem-solving in language models integrating symbolic formulation and self-refinement.

Output:
Title: Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning

Abstract: Large Language Models (LLMs) have shown human-like reasoning abilities but still struggle with complex logical problems. This paper introduces a novel framework, Logic-LM, which integrates LLMs with symbolic solvers to improve logical problem-solving. Our method first utilizes LLMs to translate a natural language problem into a symbolic formulation. Afterward, a deterministic symbolic solver performs inference on the formulated problem. We also introduce a self-refinement module, which utilizes the symbolic solver's error messages to revise symbolic formalizations. We demonstrate Logic-LM's effectiveness on five logical reasoning datasets: ProofWriter, PrOntoQA, FOLIO, LogicalDeduction, and AR-LSAT. On average, Logic-LM achieves a significant performance boost of 39.2% over using LLM alone with standard prompting and 18.4% over LLM with chain-of-thought prompting. Our findings suggest that Logic-LM, by combining LLMs with symbolic logic, offers a promising avenue for faithful logical reasoning. Code and data are publicly available at https://github.com/teacherpeterpan/Logic-LLM.


Prompt: Looking for the paper on the 2020 CLEF edition covering automatic claim verification in social media.

Output: Looking for the paper on the 2020 CLEF edition covering automatic claim verification in social media.

Output:
Title: Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media

Abstract: We present an overview of the third edition of the CheckThat! Lab at CLEF 2020. The lab featured five tasks in two different languages: English and Arabic. The first four tasks compose the full pipeline of claim verification in social media: Task 1 on check-worthiness estimation, Task 2 on retrieving previously fact-checked claims, Task 3 on evidence retrieval, and Task 4 on claim verification. The lab is completed with Task 5 on check-worthiness estimation in political debates and speeches. A total of 67 teams registered to participate in the lab (up from 47 at CLEF 2019), and 23 of them actually submitted runs (compared to 14 at CLEF 2019). Most teams used deep neural networks based on BERT, LSTMs, or CNNs, and achieved sizable improvements over the baselines on all tasks. Here we describe the tasks setup, the evaluation results, and a summary of the approaches used by the participants, and we discuss some lessons learned. Last but not least, we release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in the important tasks of check-worthiness estimation and automatic claim verification.


Prompt: Looking for the paper on the 2020 CLEF edition covering automatic claim verification in social media.

Output: Looking for the paper on the 2020 CLEF edition covering automatic claim verification in social media.

Output:
Title: Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media

Abstract: We present an overview of the third edition of the CheckThat! Lab at CLEF 2020. The lab featured five tasks in two different languages: English and Arabic. The first four tasks compose the full pipeline of claim verification in social media: Task 1 on check-worthiness estimation, Task 2 on retrieving previously fact-checked claims, Task 3 on evidence retrieval, and Task 4 on claim verification. The lab is completed with Task 5 on check-worthiness estimation in political debates and speeches. A total of 67 teams registered to participate in the lab (up from 47 at CLEF 2019), and 23 of them actually submitted runs (compared to 14 at CLEF 2019). Most teams used deep neural networks based on BERT, LSTMs, or CNNs, and achieved sizable improvements over the baselines on all tasks. Here we describe the tasks setup, the evaluation results, and a summary of the approaches used by the participants, and we discuss some lessons learned. Last but not least, we release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in the important tasks of check-worthiness estimation and automatic claim verification.


Prompt: can r squared adjusted be negative?

Output: can r squared adjusted be negative?

Output:
Stats Stackexchange

Q: Is it a problem to get a negative adjusted r-squared? Background:
I have the cross-sectional model:
$Y_{i}  = a + b X_{1,i} + c X_{2,i} + d X_{3,i} + e X_{4,i} + \nu_i$.
The application is corporate finance. So each $Y_i$ is something like the change in return on assets over a 1 year period for firm $i$, and the regressors are typical corporate finance variables.
In corporate finance, very small values of $R^2$ are common, even sometimes $1\%$. My $R^2$ is around $1\%$ but my Adjusted $R^2$ is $-0.2\%$.
I have never seen papers report negative Adjusted $R^2$ but this could just be because they omit the publication of their adjusted $R^2$ when they see that it is negative.
Question
Is there some problem when the adjusted $R^2$ is negative?

A: The formula for adjusted R square allows it to be negative.  It is intended to approximate the actual percentage variance explained.  So if the actual R square is close to zero the adjusted R square can be slightly negative.  Just think of it as an estimate of zero.


Prompt: can r squared adjusted be negative?

Output: can r squared adjusted be negative?

Output:
Stats Stackexchange

Q: Is it a problem to get a negative adjusted r-squared? Background:
I have the cross-sectional model:
$Y_{i}  = a + b X_{1,i} + c X_{2,i} + d X_{3,i} + e X_{4,i} + \nu_i$.
The application is corporate finance. So each $Y_i$ is something like the change in return on assets over a 1 year period for firm $i$, and the regressors are typical corporate finance variables.
In corporate finance, very small values of $R^2$ are common, even sometimes $1\%$. My $R^2$ is around $1\%$ but my Adjusted $R^2$ is $-0.2\%$.
I have never seen papers report negative Adjusted $R^2$ but this could just be because they omit the publication of their adjusted $R^2$ when they see that it is negative.
Question
Is there some problem when the adjusted $R^2$ is negative?

A: The formula for adjusted R square allows it to be negative.  It is intended to approximate the actual percentage variance explained.  So if the actual R square is close to zero the adjusted R square can be slightly negative.  Just think of it as an estimate of zero.


Prompt: Paper showcasing reinforcement learning from human feedback to train beneficial and safe AI assistants.

Output: Paper showcasing reinforcement learning from human feedback to train beneficial and safe AI assistants.

Output:
Title: Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Abstract: We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.


Prompt: Paper showcasing reinforcement learning from human feedback to train beneficial and safe AI assistants.

Output: Paper showcasing reinforcement learning from human feedback to train beneficial and safe AI assistants.

Output:
Title: Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Abstract: We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.


Prompt: Looking for a paper on Fine-grained Entity Type Classification using a neural network with hierarchy-aware loss.

Output: Looking for a paper on Fine-grained Entity Type Classification using a neural network with hierarchy-aware loss.

Output:
Title: Graph Neural Network Approach to Semantic Type Detection in Tables

Abstract: This study addresses the challenge of detecting semantic column types in relational tables, a key task in many real-world applications. While language models like BERT have improved prediction accuracy, their token input constraints limit the simultaneous processing of intra-table and inter-table information. We propose a novel approach using Graph Neural Networks (GNNs) to model intra-table dependencies, allowing language models to focus on inter-table information. Our proposed method not only outperforms existing state-of-the-art algorithms but also offers novel insights into the utility and functionality of various GNN types for semantic type detection. The code is available at https://github.com/hoseinzadeehsan/GAIT


Prompt: Looking for a paper on Fine-grained Entity Type Classification using a neural network with hierarchy-aware loss.

Output: Looking for a paper on Fine-grained Entity Type Classification using a neural network with hierarchy-aware loss.

Output:
Title: Neural Fine-Grained Entity Type Classification with Hierarchy-Aware Loss

Abstract: The task of Fine-grained Entity Type Classification (FETC) consists of assigning types from a hierarchy to entity mentions in text. Existing methods rely on distant supervision and are thus susceptible to noisy labels that can be out-of-context or overly-specific for the training sentence. Previous methods that attempt to address these issues do so with heuristics or with the help of hand-crafted features. Instead, we propose an end-to-end solution with a neural network model that uses a variant of cross- entropy loss function to handle out-of-context labels, and hierarchical loss normalization to cope with overly-specific ones. Also, previous work solve FETC a multi-label classification followed by ad-hoc post-processing. In contrast, our solution is more elegant: we use public word embeddings to train a single-label that jointly learns representations for entity mentions and their context. We show experimentally that our approach is robust against noise and consistently outperforms the state-of-the-art on established benchmarks for the task.


Prompt: A paper introducing a new large language model

Output: A paper introducing a new large language model

Output:
Title: Larger-Context Language Modelling

Abstract: In this work, we propose a novel method to incorporate corpus-level discourse information into language modelling. We call this larger-context language model. We introduce a late fusion approach to a recurrent language model based on long short-term memory units (LSTM), which helps the LSTM unit keep intra-sentence dependencies and inter-sentence dependencies separate from each other. Through the evaluation on three corpora (IMDB, BBC, and PennTree Bank), we demon- strate that the proposed model improves perplexity significantly. In the experi- ments, we evaluate the proposed approach while varying the number of context sentences and observe that the proposed late fusion is superior to the usual way of incorporating additional inputs to the LSTM. By analyzing the trained larger- context language model, we discover that content words, including nouns, adjec- tives and verbs, benefit most from an increasing number of context sentences. This analysis suggests that larger-context language model improves the unconditional language model by capturing the theme of a document better and more easily.


Prompt: A paper introducing a new large language model

Output: A paper introducing a new large language model

Output:
Title: Large Language Models: A Survey

Abstract: Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws \cite{kaplan2020scaling,hoffmann2022training}. The research area of LLMs, while very recent, is evolving rapidly in many different ways. In this paper, we review some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discuss their characteristics, contributions and limitations. We also give an overview of techniques developed to build, and augment LLMs. We then survey popular datasets prepared for LLM training, fine-tuning, and evaluation, review widely used LLM evaluation metrics, and compare the performance of several popular LLMs on a set of representative benchmarks. Finally, we conclude the paper by discussing open challenges and future research directions.


Prompt: How can BERT models be adapted to perform better in biomedical natural language processing?

Output: How can BERT models be adapted to perform better in biomedical natural language processing?

Output:
Title: Spatiotemporal complexity of a ratio-dependent predator-prey system

Abstract: In this paper, we investigate the emergence of a ratio-dependent predator-prey system with Michaelis-Menten-type functional response and reaction-diffusion. We derive the conditions for Hopf, Turing and Wave bifurcation on a spatial domain. Furthermore, we present a theoretical analysis of evolutionary processes that involves organisms distribution and their interaction of spatially distributed population with local diffusion. The results of numerical simulations reveal that the typical dynamics of population density variation is the formation of isolated groups, i.e., stripelike or spotted or coexistence of both. Our study shows that the spatially extended model has not only more complex dynamic patterns in the space, but also chaos and spiral waves. It may help us better understand the dynamics of an aquatic community in a real marine environment.


Prompt: How can BERT models be adapted to perform better in biomedical natural language processing?

Output: How can BERT models be adapted to perform better in biomedical natural language processing?

Output:
Title: Absorbing processes in Richardson diffusion: analytical results

Abstract: We consider the recently addressed problem of a passive particle (a predator), being the center of a ``sphere of interception'' of radius $R$ and able to absorb other passive particles (the preys) entering into the sphere. Assuming that all the particles are advected by a turbulent flow and that, in particular, the Richardson equation properly describes the relative dispersion, we calculate an analytical expression for the flux into the sphere as a function of time, assuming an initial constant density of preys outside the sphere. In the same framework, we show that the distribution of times of first passage into the sphere has a $t^{-5/2}$ power law tail, seen in contrast to the $t^{-3/2}$ appearing in standard 3D diffusion. We also discuss the correction due to the integral length scale on the results in the stationary case.


Prompt: Best practices and taxonomy of data selection for efficient language model pre-training.

Output: Best practices and taxonomy of data selection for efficient language model pre-training.

Output:
Title: How to Train Data-Efficient LLMs

Abstract: The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we propose Density sampling, which models the data distribution to select a diverse sample. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories. Coverage sampling can recover the performance of the full data, while models trained on Ask-LLM data consistently outperform full-data training -- even when we reject 90% of the original dataset, while converging up to 70% faster.


Prompt: Best practices and taxonomy of data selection for efficient language model pre-training.

Output: Best practices and taxonomy of data selection for efficient language model pre-training.

Output:
Title: A Survey on Data Selection for Language Models

Abstract: A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training. However, naively training a model on all available data may not be optimal (or feasible), as the quality of available text data can vary. Filtering out data can also decrease the carbon footprint and financial costs of training models by reducing the amount of training required. Data selection methods aim to determine which candidate data points to include in the training dataset and how to appropriately sample from the selected data points. The promise of improved data selection methods has caused the volume of research in the area to rapidly expand. However, because deep learning is mostly driven by empirical evidence and experimentation on large-scale data is expensive, few organizations have the resources for extensive data selection research. Consequently, knowledge of effective data selection practices has become concentrated within a few organizations, many of which do not openly share their findings and methodologies. To narrow this gap in knowledge, we present a comprehensive review of existing literature on data selection methods and related research areas, providing a taxonomy of existing approaches. By describing the current landscape of research, this work aims to accelerate progress in data selection by establishing an entry point for new and established researchers. Additionally, throughout this review we draw attention to noticeable holes in the literature and conclude the paper by proposing promising avenues for future research.


Prompt: Paper detailing advancements in self-supervised learning with ResNets that outperforms supervised benchmarks using ReLICv2.

Output: Paper detailing advancements in self-supervised learning with ResNets that outperforms supervised benchmarks using ReLICv2.

Output:
Title: Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?

Abstract: Despite recent progress made by self-supervised methods in representation learning with residual networks, they still underperform supervised learning on the ImageNet classification benchmark, limiting their applicability in performance-critical settings. Building on prior theoretical insights from ReLIC [Mitrovic et al., 2021], we include additional inductive biases into self-supervised learning. We propose a new self-supervised representation learning method, ReLICv2, which combines an explicit invariance loss with a contrastive objective over a varied set of appropriately constructed data views to avoid learning spurious correlations and obtain more informative representations. ReLICv2 achieves $77.1\%$ top-$1$ accuracy on ImageNet under linear evaluation on a ResNet50, thus improving the previous state-of-the-art by absolute $+1.5\%$; on larger ResNet models, ReLICv2 achieves up to $80.6\%$ outperforming previous self-supervised approaches with margins up to $+2.3\%$. Most notably, ReLICv2 is the first unsupervised representation learning method to consistently outperform the supervised baseline in a like-for-like comparison over a range of ResNet architectures. Using ReLICv2, we also learn more robust and transferable representations that generalize better out-of-distribution than previous work, both on image classification and semantic segmentation. Finally, we show that despite using ResNet encoders, ReLICv2 is comparable to state-of-the-art self-supervised vision transformers.


Prompt: Paper detailing advancements in self-supervised learning with ResNets that outperforms supervised benchmarks using ReLICv2.

Output: Paper detailing advancements in self-supervised learning with ResNets that outperforms supervised benchmarks using ReLICv2.

Output:
Title: A Recharge Oscillator Model for Interannual Variability in Venus' Clouds

Abstract: Sulfur dioxide is a radiatively and chemically important trace gas in the atmosphere of Venus and its abundance at the cloud-tops has been observed to vary on interannual to decadal timescales. This variability is thought to come from changes in the strength of convection which transports sulfur dioxide to the cloud-tops, {although} the dynamics behind such convective variability are unknown. Here we propose a new conceptual model for convective variability that links the radiative effects of water abundance at the cloud-base to convective strength within the clouds, which in turn affects water transport within the cloud. The model consists of two coupled equations which are identified as a recharge-discharge oscillator. The solutions of the coupled equations are finite amplitude sustained oscillations in convective strength and cloud-base water abundance on 3-9 year timescales. The characteristic oscillation timescale is given by the geometric mean of the radiative cooling time and the eddy mixing time near the base of the convective clouds.