codefuse-ai

/

F2LLM-v2-14B

Like 21

Follow

CodeFuse AI 245

Is this a dual-encoder architecture? From the paper it looks like it is not

The F2LLM families are all general-purpose embedding models, and hence dual-encoder in the context of retrieval.

Does the output vector support MRL (Matryoshka Representation Learning)? It looks like yes, but this info should stay in the model card

Yes, MRL is supported. Thanks for the suggestion. We will add it to the model cards.

How long is the context window (max number of tokens)? Without this, the model cannot be used at all

The models use Rotary Position Embedding (RoPE), allowing them to generalize to input context beyond that used during training. They are trained with a maximum input length of 2048 and support a max context length of 40960 in the current configuration. In practice, we recommend truncating the input to a shorter length (such as 8192) for throughput considerations.

Thank you @Geralt-TargarYen .

The F2LLM families are all general-purpose embedding models, and hence dual-encoder in the context of retrieval.

I was referring to the Qwen embeddings that have been trained so that questions and their answers are closer.

The models use Rotary Position Embedding (RoPE), allowing them to generalize to input context beyond that used during training. They are trained with a maximum input length of 2048 and support a max context length of 40960 in the current configuration. In practice, we recommend truncating the input to a shorter length (such as 8192) for throughput considerations.

Do you mean that we should clear the model context entirely before requesting embeddings for a totally different document? This information is very important as it changes entirely the strategy in ingesting the information.

Best,

Yes, the query and each document should be encoded independently. You can refer to the example in the model card for detail.

Yes, the query and each document should be encoded independently. You can refer to the example in the model card for detail.

Of course, this always happens as you first compute the documents embedding and later on you will have to find them through the query.
My question is different: Qwen models are specifically trained to make "What is the capital of China?" and "The capital of China is Beijing." close to each other. Traditional embedding models are instead trained to search another query like "What is the most enjoyable capital in Europe".

Thank you

I think you are referring to two different types of commonly used data. A query "What is the capital of China?" with a document "The capital of China is Beijing." is an instance of retrieval data, while the same query with the document "What is the most enjoyable capital in Europe" is an instance of STS (semantic textual similarity) data. Both types of data are abundant in our training dataset (https://huggingface.co/datasets/codefuse-ai/F2LLM-v2). We distinguish them via different instructions.

Got it, thank you. You approach the problem with prompting rather than specializing the model as Qwen3 did.