Prompt: Study showing improvements in spatio-temporal video grounding with transformer-based methods

Output: Study showing improvements in spatio-temporal video grounding with transformer-based methods

Output:
Title: Grounding Spatio-Temporal Language with Transformers

Abstract: Language is an interface to the outside world. In order for embodied agents to use it, language must be grounded in other, sensorimotor modalities. While there is an extended literature studying how machines can learn grounded language, the topic of how to learn spatio-temporal linguistic concepts is still largely uncharted. To make progress in this direction, we here introduce a novel spatio-temporal language grounding task where the goal is to learn the meaning of spatio-temporal descriptions of behavioral traces of an embodied agent. This is achieved by training a truth function that predicts if a description matches a given history of observations. The descriptions involve time-extended predicates in past and present tense as well as spatio-temporal references to objects in the scene. To study the role of architectural biases in this task, we train several models including multimodal Transformer architectures; the latter implement different attention computations between words and objects across space and time. We test models on two classes of generalization: 1) generalization to randomly held-out sentences; 2) generalization to grammar primitives. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall, and that summarizing object traces in a single token has little influence on performance. We then discuss how this opens new perspectives for language-guided autonomous embodied agents. We also release our code under open-source license as well as pretrained models and datasets to encourage the wider community to build upon and extend our work in the future.


Prompt: A model improving question generation and answering with Transformers in semi-supervised setup.

Output: A model improving question generation and answering with Transformers in semi-supervised setup.

Output:
Title: Machine Comprehension by Text-to-Text Neural Question Generation

Abstract: We propose a recurrent neural model that generates natural-language questions from documents, conditioned on answers. We show how to train the model using a combination of supervised and reinforcement learning. After teacher forcing for standard maximum likelihood training, we fine-tune the model using policy gradient techniques to maximize several rewards that measure question quality. Most notably, one of these rewards is the performance of a question-answering system. We motivate question generation as a means to improve the performance of question answering systems. Our model is trained and evaluated on the recent question-answering dataset SQuAD.


Prompt: A model improving question generation and answering with Transformers in semi-supervised setup.

Output: A model improving question generation and answering with Transformers in semi-supervised setup.

Output:
Title: Hexagons and Interfaces in a Vibrated Granular Layer

Abstract: The order parameter model based on parametric Ginzburg-Landau equation is used to describe high acceleration patterns in vibrated layer of granular material. At large amplitude of driving both hexagons and interfaces emerge. Transverse instability leading to formation of ``decorated'' interfaces and labyrinthine patterns, is found. Additional sub-harmonic forcing leads to controlled interface motion.


Prompt: A model replacing RNN/CNN with self-attention for better HAR results

Output: A model replacing RNN/CNN with self-attention for better HAR results

Output:
Title: Partial Representations and Amenable Fell Bundles over Free Groups

Abstract: We show that a Fell bundle B = {B_t}_{t \in F}, over an arbitrary free group F, is amenable, whenever it is orthogonal (in the sense that B_x^* B_y = 0, if x and y are distinct generators of F) and semi-saturated (in the sense that B_{ts} coincides with the closed linear span of B_t B_s, when the multiplication ``ts'' involves no cancelation).


Prompt: A model replacing RNN/CNN with self-attention for better HAR results

Output: A model replacing RNN/CNN with self-attention for better HAR results

Output:
Title: Electromagnetic transitions with effective operators

Abstract: In the no-core shell model formalism we compute effective one- and two-body operators, using the Lee-Suzuki procedure within the two-body cluster approximation. We evaluate the validity of the latter through calculations in reduced model spaces. In particular, we test the results for the two-body system and find that indeed the effective operators in the reduced space reproduce the expectation values or transition strengths computed in the full space. On the other hand, the renormalization for operators in the case of 6Li is very weak, suggesting the need for higher-body clusters in computing the effective interaction.


Prompt: A comprehensive study on an autoregressive language model that achieves competitive results without fine-tuning on translation, question-answering, and other tasks.

Output: A comprehensive study on an autoregressive language model that achieves competitive results without fine-tuning on translation, question-answering, and other tasks.

Output:
Title: A Comprehensive Performance Study of Large Language Models on Novel AI Accelerators

Abstract: Artificial intelligence (AI) methods have become critical in scientific applications to help accelerate scientific discovery. Large language models (LLMs) are being considered as a promising approach to address some of the challenging problems because of their superior generalization capabilities across domains. The effectiveness of the models and the accuracy of the applications is contingent upon their efficient execution on the underlying hardware infrastructure. Specialized AI accelerator hardware systems have recently become available for accelerating AI applications. However, the comparative performance of these AI accelerators on large language models has not been previously studied. In this paper, we systematically study LLMs on multiple AI accelerators and GPUs and evaluate their performance characteristics for these models. We evaluate these systems with (i) a micro-benchmark using a core transformer block, (ii) a GPT- 2 model, and (iii) an LLM-driven science use case, GenSLM. We present our findings and analyses of the models' performance to better understand the intrinsic capabilities of AI accelerators. Furthermore, our analysis takes into account key factors such as sequence lengths, scaling behavior, sparsity, and sensitivity to gradient accumulation steps.


Prompt: A comprehensive study on an autoregressive language model that achieves competitive results without fine-tuning on translation, question-answering, and other tasks.

Output: A comprehensive study on an autoregressive language model that achieves competitive results without fine-tuning on translation, question-answering, and other tasks.

Output:
Title: What Have We Achieved on Non-autoregressive Translation?

Abstract: Recent advances have made non-autoregressive (NAT) translation comparable to autoregressive methods (AT). However, their evaluation using BLEU has been shown to weakly correlate with human annotations. Limited research compares non-autoregressive translation and autoregressive translation comprehensively, leaving uncertainty about the true proximity of NAT to AT. To address this gap, we systematically evaluate four representative NAT methods across various dimensions, including human evaluation. Our empirical results demonstrate that despite narrowing the performance gap, state-of-the-art NAT still underperforms AT under more reliable evaluation metrics. Furthermore, we discover that explicitly modeling dependencies is crucial for generating natural language and generalizing to out-of-distribution sequences.


Prompt: Seeking a paper on refining model predictions layer-by-layer in transformers with improved techniques.

Output: Seeking a paper on refining model predictions layer-by-layer in transformers with improved techniques.

Output:
Title: Eliciting Latent Predictions from Transformers with the Tuned Lens

Abstract: We analyze transformers from the perspective of iterative inference, seeking to understand how model predictions are refined layer by layer. To do so, we train an affine probe for each block in a frozen pretrained model, making it possible to decode every hidden state into a distribution over the vocabulary. Our method, the \emph{tuned lens}, is a refinement of the earlier ``logit lens'' technique, which yielded useful insights but is often brittle. We test our method on various autoregressive language models with up to 20B parameters, showing it to be more predictive, reliable and unbiased than the logit lens. With causal experiments, we show the tuned lens uses similar features to the model itself. We also find the trajectory of latent predictions can be used to detect malicious inputs with high accuracy. All code needed to reproduce our results can be found at https://github.com/AlignmentResearch/tuned-lens.


Prompt: Seeking a paper on refining model predictions layer-by-layer in transformers with improved techniques.

Output: Seeking a paper on refining model predictions layer-by-layer in transformers with improved techniques.

Output:
Title: Rewiring the Transformer with Depth-Wise LSTMs

Abstract: Stacking non-linear layers allows deep neural networks to model complicated functions, and including residual connections in Transformer layers is beneficial for convergence and performance. However, residual connections may make the model "forget" distant layers and fail to fuse information from previous layers effectively. Selectively managing the representation aggregation of Transformer layers may lead to better performance. In this paper, we present a Transformer with depth-wise LSTMs connecting cascading Transformer layers and sub-layers. We show that layer normalization and feed-forward computation within a Transformer layer can be absorbed into depth-wise LSTMs connecting pure Transformer attention layers. Our experiments with the 6-layer Transformer show significant BLEU improvements in both WMT 14 English-German / French tasks and the OPUS-100 many-to-many multilingual NMT task, and our deep Transformer experiments demonstrate the effectiveness of depth-wise LSTM on the convergence and performance of deep Transformers.


Prompt: A model improving the accuracy of location-based recommendations by 9-17% through relative spatiotemporal correlations.

Output: A model improving the accuracy of location-based recommendations by 9-17% through relative spatiotemporal correlations.

Output:
Title: A Joint Two-Phase Time-Sensitive Regularized Collaborative Ranking Model for Point of Interest Recommendation

Abstract: The popularity of location-based social networks (LBSNs) has led to a tremendous amount of user check-in data. Recommending points of interest (POIs) plays a key role in satisfying users' needs in LBSNs. While recent work has explored the idea of adopting collaborative ranking (CR) for recommendation, there have been few attempts to incorporate temporal information for POI recommendation using CR. In this article, we propose a two-phase CR algorithm that incorporates the geographical influence of POIs and is regularized based on the variance of POIs popularity and users' activities over time. The time-sensitive regularizer penalizes user and POIs that have been more time-sensitive in the past, helping the model to account for their long-term behavioral patterns while learning from user-POI interactions. Moreover, in the first phase, it attempts to rank visited POIs higher than the unvisited ones, and at the same time, apply the geographical influence. In the second phase, our algorithm tries to rank users' favorite POIs higher on the recommendation list. Both phases employ a collaborative learning strategy that enables the model to capture complex latent associations from two different perspectives. Experiments on real-world datasets show that our proposed time-sensitive collaborative ranking model beats state-of-the-art POI recommendation methods.


Prompt: A model improving the accuracy of location-based recommendations by 9-17% through relative spatiotemporal correlations.

Output: A model improving the accuracy of location-based recommendations by 9-17% through relative spatiotemporal correlations.

Output:
Title: Memory efficient location recommendation through proximity-aware representation

Abstract: Sequential location recommendation plays a huge role in modern life, which can enhance user experience, bring more profit to businesses and assist in government administration. Although methods for location recommendation have evolved significantly thanks to the development of recommendation systems, there is still limited utilization of geographic information, along with the ongoing challenge of addressing data sparsity. In response, we introduce a Proximity-aware based region representation for Sequential Recommendation (PASR for short), built upon the Self-Attention Network architecture. We tackle the sparsity issue through a novel loss function employing importance sampling, which emphasizes informative negative samples during optimization. Moreover, PASR enhances the integration of geographic information by employing a self-attention-based geography encoder to the hierarchical grid and proximity grid at each GPS point. To further leverage geographic information, we utilize the proximity-aware negative samplers to enhance the quality of negative samples. We conducted evaluations using three real-world Location-Based Social Networking (LBSN) datasets, demonstrating that PASR surpasses state-of-the-art sequential location recommendation methods


Prompt: can you travel to kenya with an expired passport?

Output: can you travel to kenya with an expired passport?

Output:
Gaming Stackexchange

Q: Can I get a document back after I have returned it? After approving a migrant worker's passport and returning it to them, I realized that it was expired. I still have their work ticket, which appears valid. Is there any way to deny them entry to the country at this point?

A: Nope.
Well, you just made a big mistake: You approved before checking everything. The game is very harsh in this regard and does not allow you to roll back your decision, once you have given back the document.
This is part of the message - Think before you act, even if it takes valuable time ... Or just push your luck.
Also: You don't get fined immediately for a simple error - you get two chances every day to make mistakes (intentional or otherwise), and then the fines start kicking in.


Prompt: can you travel to kenya with an expired passport?

Output: can you travel to kenya with an expired passport?

Output:
Gaming Stackexchange

Q: Can I get a document back after I have returned it? After approving a migrant worker's passport and returning it to them, I realized that it was expired. I still have their work ticket, which appears valid. Is there any way to deny them entry to the country at this point?

A: Nope.
Well, you just made a big mistake: You approved before checking everything. The game is very harsh in this regard and does not allow you to roll back your decision, once you have given back the document.
This is part of the message - Think before you act, even if it takes valuable time ... Or just push your luck.
Also: You don't get fined immediately for a simple error - you get two chances every day to make mistakes (intentional or otherwise), and then the fines start kicking in.


Prompt: Paper on GPt models for embedding

Output: Paper on GPt models for embedding

Output:
Title: Gaussian Process Topic Models

Abstract: We introduce Gaussian Process Topic Models (GPTMs), a new family of topic models which can leverage a kernel among documents while extracting correlated topics. GPTMs can be considered a systematic generalization of the Correlated Topic Models (CTMs) using ideas from Gaussian Process (GP) based embedding. Since GPTMs work with both a topic covariance matrix and a document kernel matrix, learning GPTMs involves a novel component-solving a suitable Sylvester equation capturing both topic and document dependencies. The efficacy of GPTMs is demonstrated with experiments evaluating the quality of both topic modeling and embedding.


Prompt: Paper on GPt models for embedding

Output: Paper on GPt models for embedding

Output:
Title: Nimble GNN Embedding with Tensor-Train Decomposition

Abstract: This paper describes a new method for representing embedding tables of graph neural networks (GNNs) more compactly via tensor-train (TT) decomposition. We consider the scenario where (a) the graph data that lack node features, thereby requiring the learning of embeddings during training; and (b) we wish to exploit GPU platforms, where smaller tables are needed to reduce host-to-GPU communication even for large-memory GPUs. The use of TT enables a compact parameterization of the embedding, rendering it small enough to fit entirely on modern GPUs even for massive graphs. When combined with judicious schemes for initialization and hierarchical graph partitioning, this approach can reduce the size of node embedding vectors by 1,659 times to 81,362 times on large publicly available benchmark datasets, achieving comparable or better accuracy and significant speedups on multi-GPU systems. In some cases, our model without explicit node features on input can even match the accuracy of models that use node features.


Prompt: Work using GPT models for text embeddings

Output: Work using GPT models for text embeddings

Output:
Title: SGPT: GPT Sentence Embeddings for Semantic Search

Abstract: Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.


Prompt: Work using GPT models for text embeddings

Output: Work using GPT models for text embeddings

Output:
Title: SGPT: GPT Sentence Embeddings for Semantic Search

Abstract: Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.


Prompt: How to use LLMs for text embedding?

Output: How to use LLMs for text embedding?

Output:
Title: Subgroups of inertia groups arising from abelian varieties

Abstract: Given an abelian variety over a field with a discrete valuation, Grothendieck defined a certain open normal subgroup of the absolute inertia group. This subgroup encodes information on the extensions over which the abelian variety acquires semistable reduction. We study this subgroup, and use it to obtain information on the extensions over which the abelian variety acquires semistable reduction.


Prompt: How to use LLMs for text embedding?

Output: How to use LLMs for text embedding?

Output:
Title: Polarizability Extraction for Waveguide-Fed Metasurfaces

Abstract: We consider the design and modeling of metasurfaces that couple energy from guided waves to propagating wavefronts. This is a first step towards a comprehensive, multiscale modeling platform for metasurface antennas-large arrays of metamaterial elements embedded in a waveguide structure that radiates into free-space--in which the detailed electromagnetic responses of metamaterial elements are replaced by polarizable dipoles. We present two methods to extract the effective polarizability of a metamaterial element embedded in a one- or two-dimensional waveguide. The first method invokes surface equivalence principles, averaging over the effective surface currents and charges within an element to obtain the effective dipole moments; the second method is based on computing the coefficients of the scattered waves within the waveguide, from which the effective polarizability can be inferred. We demonstrate these methods on several variants of waveguide-fed metasurface elements, finding excellent agreement between the two, as well as with analytical expressions derived for irises with simpler geometries. Extending the polarizability extraction technique to higher order multipoles, we confirm the validity of the dipole approximation for common metamaterial elements. With the effective polarizabilities of the metamaterial elements accurately determined, the radiated fields generated by a metasurface antenna (inside and outside the antenna) can be found self-consistently by including the interactions between polarizable dipoles. The dipole description provides an alternative language and computational framework for engineering metasurface antennas, holograms, lenses, beam-forming arrays, and other electrically large, waveguide-fed metasurface structures.


Prompt: How to use LLMs for text embedding?

Output: How to use LLMs for text embedding?

Output:
Title: Text clustering with LLM embeddings

Abstract: Text clustering is an important approach for organising the growing amount of digital content, helping to structure and find hidden patterns in uncategorised data. However, the effectiveness of text clustering heavily relies on the choice of textual embeddings and clustering algorithms. We argue that recent advances in large language models (LLMs) can potentially improve this task. In this research, we investigated how different textual embeddings -- particularly those used in LLMs -- and clustering algorithms affect how text datasets are clustered. A series of experiments were conducted to assess how embeddings influence clustering results, the role played by dimensionality reduction through summarisation, and model size adjustment. Findings reveal that LLM embeddings excel at capturing subtleties in structured language, while BERT leads the lightweight options in performance. In addition, we observe that increasing model dimensionality and employing summarization techniques do not consistently lead to improvements in clustering efficiency, suggesting that these strategies require careful analysis to use in real-life models. These results highlight a complex balance between the need for refined text representation and computational feasibility in text clustering applications. This study extends traditional text clustering frameworks by incorporating embeddings from LLMs, providing a path for improved methodologies, while informing new avenues for future research in various types of textual analysis.


Prompt: How to use LLMs for text embedding?

Output: How to use LLMs for text embedding?

Output:
Title: Improving Text Embeddings with Large Language Models

Abstract: In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods that often depend on multi-stage intermediate pre-training with billions of weakly-supervised text pairs, followed by fine-tuning with a few labeled datasets, our method does not require building complex training pipelines or relying on manually collected datasets that are often constrained by task diversity and language coverage. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. We then fine-tune open-source decoder-only LLMs on the synthetic data using standard contrastive loss. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data. Furthermore, when fine-tuned with a mixture of synthetic and labeled data, our model sets new state-of-the-art results on the BEIR and MTEB benchmarks.


Prompt: How to use LLMs for text embedding?

Output: How to use LLMs for text embedding?

Output:
Title: Text clustering with LLM embeddings

Abstract: Text clustering is an important approach for organising the growing amount of digital content, helping to structure and find hidden patterns in uncategorised data. However, the effectiveness of text clustering heavily relies on the choice of textual embeddings and clustering algorithms. We argue that recent advances in large language models (LLMs) can potentially improve this task. In this research, we investigated how different textual embeddings -- particularly those used in LLMs -- and clustering algorithms affect how text datasets are clustered. A series of experiments were conducted to assess how embeddings influence clustering results, the role played by dimensionality reduction through summarisation, and model size adjustment. Findings reveal that LLM embeddings excel at capturing subtleties in structured language, while BERT leads the lightweight options in performance. In addition, we observe that increasing model dimensionality and employing summarization techniques do not consistently lead to improvements in clustering efficiency, suggesting that these strategies require careful analysis to use in real-life models. These results highlight a complex balance between the need for refined text representation and computational feasibility in text clustering applications. This study extends traditional text clustering frameworks by incorporating embeddings from LLMs, providing a path for improved methodologies, while informing new avenues for future research in various types of textual analysis.


Prompt: How to use LLMs for text embedding?

Output: How to use LLMs for text embedding?

Output:
Title: Polarizability Extraction for Waveguide-Fed Metasurfaces

Abstract: We consider the design and modeling of metasurfaces that couple energy from guided waves to propagating wavefronts. This is a first step towards a comprehensive, multiscale modeling platform for metasurface antennas-large arrays of metamaterial elements embedded in a waveguide structure that radiates into free-space--in which the detailed electromagnetic responses of metamaterial elements are replaced by polarizable dipoles. We present two methods to extract the effective polarizability of a metamaterial element embedded in a one- or two-dimensional waveguide. The first method invokes surface equivalence principles, averaging over the effective surface currents and charges within an element to obtain the effective dipole moments; the second method is based on computing the coefficients of the scattered waves within the waveguide, from which the effective polarizability can be inferred. We demonstrate these methods on several variants of waveguide-fed metasurface elements, finding excellent agreement between the two, as well as with analytical expressions derived for irises with simpler geometries. Extending the polarizability extraction technique to higher order multipoles, we confirm the validity of the dipole approximation for common metamaterial elements. With the effective polarizabilities of the metamaterial elements accurately determined, the radiated fields generated by a metasurface antenna (inside and outside the antenna) can be found self-consistently by including the interactions between polarizable dipoles. The dipole description provides an alternative language and computational framework for engineering metasurface antennas, holograms, lenses, beam-forming arrays, and other electrically large, waveguide-fed metasurface structures.


Prompt: How to use LLMs for text embedding?

Output: How to use LLMs for text embedding?

Output:
Title: Text clustering with LLM embeddings

Abstract: Text clustering is an important approach for organising the growing amount of digital content, helping to structure and find hidden patterns in uncategorised data. However, the effectiveness of text clustering heavily relies on the choice of textual embeddings and clustering algorithms. We argue that recent advances in large language models (LLMs) can potentially improve this task. In this research, we investigated how different textual embeddings -- particularly those used in LLMs -- and clustering algorithms affect how text datasets are clustered. A series of experiments were conducted to assess how embeddings influence clustering results, the role played by dimensionality reduction through summarisation, and model size adjustment. Findings reveal that LLM embeddings excel at capturing subtleties in structured language, while BERT leads the lightweight options in performance. In addition, we observe that increasing model dimensionality and employing summarization techniques do not consistently lead to improvements in clustering efficiency, suggesting that these strategies require careful analysis to use in real-life models. These results highlight a complex balance between the need for refined text representation and computational feasibility in text clustering applications. This study extends traditional text clustering frameworks by incorporating embeddings from LLMs, providing a path for improved methodologies, while informing new avenues for future research in various types of textual analysis.


Prompt: How to use LLMs for text embedding?

Output: How to use LLMs for text embedding?

Output:
Title: How to use LLMs for Text Analysis

Abstract: This guide introduces Large Language Models (LLM) as a highly versatile text analysis method within the social sciences. As LLMs are easy-to-use, cheap, fast, and applicable on a broad range of text analysis tasks, ranging from text annotation and classification to sentiment analysis and critical discourse analysis, many scholars believe that LLMs will transform how we do text analysis. This how-to guide is aimed at students and researchers with limited programming experience, and offers a simple introduction to how LLMs can be used for text analysis in your own research project, as well as advice on best practices. We will go through each of the steps of analyzing textual data with LLMs using Python: installing the software, setting up the API, loading the data, developing an analysis prompt, analyzing the text, and validating the results. As an illustrative example, we will use the challenging task of identifying populism in political texts, and show how LLMs move beyond the existing state-of-the-art.


Prompt: How to use LLMs for text embedding?

Output: How to use LLMs for text embedding?

Output:
Title: A Robust Parser Based on Syntactic Information

Abstract: In this paper, we propose a robust parser which can parse extragrammatical sentences. This parser can recover them using only syntactic information. It can be easily modified and extended because it utilize only syntactic information.


Prompt: How to use LLMs for text embedding?

Output: How to use LLMs for text embedding?

Output:
Title: LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

Abstract: Large decoder-only language models (LLMs) are the state-of-the-art models on most of today's NLP tasks and benchmarks. Yet, the community is only slowly adopting these models for text embedding tasks, which require rich contextualized representations. In this work, we introduce LLM2Vec, a simple unsupervised approach that can transform any decoder-only LLM into a strong text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. We demonstrate the effectiveness of LLM2Vec by applying it to 3 popular LLMs ranging from 1.3B to 7B parameters and evaluate the transformed models on English word- and sequence-level tasks. We outperform encoder-only models by a large margin on word-level tasks and reach a new unsupervised state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB). Moreover, when combining LLM2Vec with supervised contrastive learning, we achieve state-of-the-art performance on MTEB among models that train only on publicly available data. Our strong empirical results and extensive analysis demonstrate that LLMs can be effectively transformed into universal text encoders in a parameter-efficient manner without the need for expensive adaptation or synthetic GPT-4 generated data.


Prompt: How to use GPT-3 for text embedding?

Output: How to use GPT-3 for text embedding?

Output:
Title: Prior Art Search and Reranking for Generated Patent Text

Abstract: Generative models, such as GPT-2, have demonstrated impressive results recently. A fundamental question we'd like to address is: where did the generated text come from? This work is our initial effort toward answering the question by using prior art search. The purpose of the prior art search is to find the most similar prior text in the training data of GPT-2. We take a reranking approach and apply it to the patent domain. Specifically, we pre-train GPT-2 models from scratch by using the patent data from the USPTO. The input for the prior art search is the patent text generated by the GPT-2 model. We also pre-trained BERT models from scratch for converting patent text to embeddings. The steps of reranking are: (1) search the most similar text in the training data of GPT-2 by taking a bag-of-word ranking approach (BM25), (2) convert the search results in text format to BERT embeddings, and (3) provide the final result by ranking the BERT embeddings based on their similarities with the patent text generated by GPT-2. The experiments in this work show that such reranking is better than ranking with embeddings alone. However, our mixed results also indicate that calculating the semantic similarities among long text spans is still challenging. To our knowledge, this work is the first to implement a reranking system to identify retrospectively the most similar inputs to a GPT model based on its output.


Prompt: How to use GPT-3 for text embedding?

Output: How to use GPT-3 for text embedding?

Output:
Title: Towards General Text Embeddings with Multi-stage Contrastive Learning

Abstract: We present GTE, a general-purpose text embedding model trained with multi-stage contrastive learning. In line with recent advancements in unifying various NLP tasks into a single format, we train a unified text embedding model by employing contrastive learning over a diverse mixture of datasets from multiple sources. By significantly increasing the number of training data during both unsupervised pre-training and supervised fine-tuning stages, we achieve substantial performance gains over existing embedding models. Notably, even with a relatively modest parameter count of 110M, GTE$_\text{base}$ outperforms the black-box embedding API provided by OpenAI and even surpasses 10x larger text embedding models on the massive text embedding benchmark. Furthermore, without additional fine-tuning on each programming language individually, our model outperforms previous best code retrievers of similar size by treating code as text. In summary, our model achieves impressive results by effectively harnessing multi-stage contrastive learning, offering a powerful and efficient text embedding model with broad applicability across various NLP and code-related tasks.


Prompt: How to use GPT for text embedding?

Output: How to use GPT for text embedding?

Output:
Title: Towards General Text Embeddings with Multi-stage Contrastive Learning

Abstract: We present GTE, a general-purpose text embedding model trained with multi-stage contrastive learning. In line with recent advancements in unifying various NLP tasks into a single format, we train a unified text embedding model by employing contrastive learning over a diverse mixture of datasets from multiple sources. By significantly increasing the number of training data during both unsupervised pre-training and supervised fine-tuning stages, we achieve substantial performance gains over existing embedding models. Notably, even with a relatively modest parameter count of 110M, GTE$_\text{base}$ outperforms the black-box embedding API provided by OpenAI and even surpasses 10x larger text embedding models on the massive text embedding benchmark. Furthermore, without additional fine-tuning on each programming language individually, our model outperforms previous best code retrievers of similar size by treating code as text. In summary, our model achieves impressive results by effectively harnessing multi-stage contrastive learning, offering a powerful and efficient text embedding model with broad applicability across various NLP and code-related tasks.


Prompt: How to use GPT for text embedding?

Output: How to use GPT for text embedding?

Output:
Title: Phase Space Derivation of a Variational Principle for One Dimensional Hamiltonian Systems

Abstract: We consider the bifurcation problem u'' + \lambda u = N(u) with two point boundary conditions where N(u) is a general nonlinear term which may also depend on the eigenvalue \lambda. A new derivation of a variational principle for the lowest eigenvalue \lambda is given. This derivation makes use only of simple algebraic inequalities and leads directly to a more explicit expression for the eigenvalue than what had been given previously.


Prompt: How to use GPT for text embedding?

Output: How to use GPT for text embedding?

Output:
Title: Numerical Criteria for vey Ampleness of Divisors on Projective Bundles over an elliptic curve

Abstract: In Butler, J.Differential Geom. 39 (1):1--34,1994, the author gives a sufficient condition for a line bundle associated with a divisor D to be normally generated on $X=P(E)$ where E is a vector bundle over a smooth curve C. A line bundle which is ample and normally generated is automatically very ample. Therefore the condition found in Butler's work, together with Miyaoka's well known ampleness criterion, give a sufficient condition for the very ampleness of D on X. This work is devoted to the study of numerical criteria for very ampleness of divisors D which do not satisfy the above criterion, in the case of C elliptic. Numerical conditions for the very ampleness of D are proved,improving existing results. In some cases a complete numerical characterization is found.


Prompt: How to use GPT for text embedding?

Output: How to use GPT for text embedding?

Output:
Title: Towards General Text Embeddings with Multi-stage Contrastive Learning

Abstract: We present GTE, a general-purpose text embedding model trained with multi-stage contrastive learning. In line with recent advancements in unifying various NLP tasks into a single format, we train a unified text embedding model by employing contrastive learning over a diverse mixture of datasets from multiple sources. By significantly increasing the number of training data during both unsupervised pre-training and supervised fine-tuning stages, we achieve substantial performance gains over existing embedding models. Notably, even with a relatively modest parameter count of 110M, GTE$_\text{base}$ outperforms the black-box embedding API provided by OpenAI and even surpasses 10x larger text embedding models on the massive text embedding benchmark. Furthermore, without additional fine-tuning on each programming language individually, our model outperforms previous best code retrievers of similar size by treating code as text. In summary, our model achieves impressive results by effectively harnessing multi-stage contrastive learning, offering a powerful and efficient text embedding model with broad applicability across various NLP and code-related tasks.


Prompt: How to use GPT for embedding & search?

Output: How to use GPT for embedding & search?

Output:
Title: Towards General Text Embeddings with Multi-stage Contrastive Learning

Abstract: We present GTE, a general-purpose text embedding model trained with multi-stage contrastive learning. In line with recent advancements in unifying various NLP tasks into a single format, we train a unified text embedding model by employing contrastive learning over a diverse mixture of datasets from multiple sources. By significantly increasing the number of training data during both unsupervised pre-training and supervised fine-tuning stages, we achieve substantial performance gains over existing embedding models. Notably, even with a relatively modest parameter count of 110M, GTE$_\text{base}$ outperforms the black-box embedding API provided by OpenAI and even surpasses 10x larger text embedding models on the massive text embedding benchmark. Furthermore, without additional fine-tuning on each programming language individually, our model outperforms previous best code retrievers of similar size by treating code as text. In summary, our model achieves impressive results by effectively harnessing multi-stage contrastive learning, offering a powerful and efficient text embedding model with broad applicability across various NLP and code-related tasks.


Prompt: How to use GPT for embedding & search?

Output: How to use GPT for embedding & search?

Output:
Title: SGPT: GPT Sentence Embeddings for Semantic Search

Abstract: Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.