This is a private model (txt)(Sorry as it is easy to extract from your own models..) but also Mistral Do sell access to thier embedding models ? I dont know why as it is in the actual 7b model or MOE models they released ?
QUESTION: Do these models contain bad embedding space as they have only been trained on random stuff and thier flagship models correctly trained on GOOD DATA, hence GuardRailing !
So just in case of ISSUES ~
MAKE A REQUEST (SORRY ALSO AS when i was uploading the files i told it to make a datasets repo but it made the normal ! so you will have top clone the repo then load_datset from Txt file::: )
Sorry as i did download the file already and split the text file with (UltraEdit) so this is mainly just an archive (for me)
Embeddings(32,000)
Embeddings are vectorial representations of text that capture the semantic meaning of paragraphs through their position in a high dimensional vector space. Mistral AI Embeddings API offers cutting-edge, state-of-the-art embeddings for text, which can be used for many NLP tasks. In the realm of text embeddings, texts with similar meanings or context tend to be located in closer proximity to each other within this space, as measured by the distance between their vectors. This is due to the fact that the model has learned to group semantically related texts together during the training process.
ALSO ::::
By using a TinyLLM you can train the llm on a single task such as what are the embedding for this term ! ie train An LLM to provide the tensor for the embedding ! instead of extracting it !
PERSONALLY :::
i have found that small slim tiny models SPecific for Purpose such as Dictionary provides a better result in downstreaming ! You may have your Super brain Query these models for such tasks ! So your instruction based CHAT! model - can stay the personality and its sub models (TINY TEAMS) can provide services such as auto agent: Hence even coding task should be sublet to smaller agents providing only this funciton : for the discussion regarding the code use the llm and the code provider use the tiny !
Extraction Processes :
As you know each model contains embeddings , here is how you can extract the mistral embeddings model (Table) to CSV : In fact it tool a while to extract (at least 2hr)
Large Filesize :
as this file was made in code ; it has only two colums :
1 The word :
- the Assosiated Tensor !
It can be easier to LookUp an embedding from a File instead of extracting them from the model which is much slower and more resource intensive: This file can be brokeni in to a split text file (1000) line eah making it easier to load into memory :
(Embeddings are not alphabetically ordered!)
Example Extraction Scrits
def lookup_embedding(file_path, term):
"""
Looks up the embedding for a given term in a text file.
Args:
file_path: The path to the text file containing term and embedding pairs.
term: The term to look up.
Returns:
A list of floats representing the embedding for the given term, or None if the term is not found.
"""
with open(file_path, 'r') as f:
for line in f:
line = line.strip()
if not line:
continue
parts = line.split()
if parts[0] == term:
return [float(x) for x in parts[1:]]
return "<unk>"
def Extract_Embedding_Vocab(model,tokenizer):
# Get the vocabulary tokens
vocab_tokens = tokenizer.get_vocab().keys()
# Convert vocabulary tokens to a list
vocab_tokens_list = list(vocab_tokens)
# Get the embeddings for each vocabulary token
embeddings_list = []
for token in vocab_tokens_list:
tokens = tokenizer(token, return_tensors="pt")
embeddings = model(**tokens).past_key_values[0][0].squeeze().tolist()
embeddings_list.append(embeddings)
# Save the vocabulary tokens and embeddings to a text file
with open(output_file_path, "w", encoding="utf-8") as output_file:
for token, embeddings in zip(vocab_tokens_list, embeddings_list):
# Write each line with token and its embedding values
output_file.write(token + "," + ",".join(map(str, embeddings)) + "\n")
output_file_path = "Mistral-7B-Instruct-v0.2_embeddings_with_vocab.txt"
print(f"Vocabulary tokens and embeddings saved to {output_file_path}")
def get_token_embedding(token,model,tokenizer):
"""
Returns the embedding for a given token using the specified BERT model.
Args:
token: A string representing the token.
Returns:
A list of floats representing the embedding.
"""
# Convert the token to its corresponding token ID
token_id = tokenizer.convert_tokens_to_ids(token)
# Create a tensor containing the token ID
input_ids = torch.tensor([token_id]).to(model.device)
# Get the model's output for the given token
model_output = model(input_ids=input_ids)
# Extract the embedding from the model's output
embedding = model_output.last_hidden_state[0][0].detach().numpy()
return embedding
def TaskGetEmbedingModel(model_name):
MODEL,TOKENIZER = LoadPretrained(model_name)
Extract_Embedding_Vocab(MODEL,TOKENIZER)
def TaskTestEmbeddingModel(model_name):
MODEL,TOKENIZER = LoadPretrained(model_name)
sample_tokens = ["the", "cat", "sat", "on", "the", "mat"]
for token in sample_tokens:
embedding = get_token_embedding(token)
print(f"Embedding for '{token}': {embedding}")
mistral example for comarison of phrases :
OR YOUR COULD USE THE API and Pay For each extraction !
```python
from mistralai.client import MistralClient
api_key = os.environ["MISTRAL_API_KEY"]
client = MistralClient(api_key=api_key)
embeddings_batch_response = client.embeddings(
model="mistral-embed",
input=["Embed this sentence.", "As well as this one."],
)
import itertools
sentences = [
"Have a safe happy Memorial Day weekend everyone",
"To all our friends at Whatsit Productions Films enjoy a safe happy Memorial Day weekend",
"Where can I find the best cheese?",
]
sentence_embeddings = [get_text_embedding(t) for t in sentences]
sentence_embeddings_pairs = list(itertools.combinations(sentence_embeddings, 2))
sentence_pairs = list(itertools.combinations(sentences, 2))
for s, e in zip(sentence_pairs, sentence_embeddings_pairs):
print(s, euclidean_distances([e[0]], [e[1]]))
- Downloads last month
- 0