Model: thanhpham1/Finetuned-all-mpnet-base-v2-with-technical-docs
Author: thanhpham1
Pipeline: sentence-similarity
Library: sentence-transformers
Tags: sentence-transformers, safetensors, mpnet, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:5146, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss, en, arxiv:1908.10084, arxiv:2205.13147, arxiv:1705.00652, base_model:sentence-transformers/all-mpnet-base-v2, base_model:finetune:sentence-transformers/all-mpnet-base-v2, license:apache-2.0, model-index, endpoints_compatible, region:us
Base models (finetune): id: sentence-transformers/all-mpnet-base-v2
Card metadata: language: en
license: apache-2.0
tags: sentence-transformers, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:5146, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss
base_model: sentence-transformers/all-mpnet-base-v2
widget: source_sentence: import subprocess
zen_of_python = subprocess.check_output(["python", "-c", "import this"])
corpus = zen_of_python.split()

num_partitions = 3
chunk = len(corpus) // num_partitions
partitions = [
corpus[i * chunk: (i + 1) * chunk] for i in range(num_partitions)
]

Mapping Data#
To determine the map phase, we require a map function to use on each document.
The output is the pair (word, 1) for every word found in a document.
For basic text documents we load as Python strings, the process is as follows:

def map_function(document):
for word in document.lower().split():
yield word, 1

We use the apply_map function on a large collection of documents by marking it as a task in Ray using the @ray.remote decorator.
When we call apply_map, we apply it to three sets of document data (num_partitions=3).
The apply_map function returns three lists, one for each partition so that Ray can rearrange the results of the map phase and distribute them to the appropriate nodes.

import ray
sentences: What does the map_function yield for each word in a document?, What does PBT do differently from traditional hyperparameter tuning methods?, What is returned by task_with_static_multiple_returns_good in the Actor class?, source_sentence: 192.168.0.15 7241 Worker ffffffffffffffffffffffffffffffffffffffff0100000001000000 10 MiB PINNED_IN_MEMORY (deserialize task arg)
__main__.f

192.168.0.15 7207 Driver ffffffffffffffffffffffffffffffffffffffff0100000001000000 15 MiB USED_BY_PENDING_TASK (put object)
test.py:
<module>:28

While the task is running, we see that ray memory shows both a LOCAL_REFERENCE and a USED_BY_PENDING_TASK reference for the object in the driver process. The worker process also holds a reference to the object because the Python arg is directly referencing the memory in the plasma, so it can’t be evicted; therefore it is PINNED_IN_MEMORY.
4. Serialized ObjectRef references
@ray.remote
def f(arg):
while True:
pass

a = ray.put(None)
b = f.remote([a])
sentences: How can a dataset be created from in-memory data?, What does Algorithm.training_step return for the new API stack?, Why can't the object be evicted while the worker process holds a reference?, source_sentence: For distributed systems engineers, Ray automatically handles key processes:

Orchestration–Managing the various components of a distributed system.
Scheduling–Coordinating when and where tasks are executed.
Fault tolerance–Ensuring tasks complete regardless of inevitable points of failure.
Auto-scaling–Adjusting the number of resources allocated to dynamic demand.


What you can do with Ray#
These are some common ML workloads that individuals, organizations, and companies leverage Ray to build their AI applications:

Batch inference on CPUs and GPUs
Model serving
Distributed training of large models
Parallel hyperparameter tuning experiments
Reinforcement learning
ML platform



Ray framework#






Stack of Ray libraries - unified toolkit for ML workloads.



Ray’s unified compute framework consists of three layers:
sentences: What does remote_worker_envs control when num_envs_per_env_runner > 1?, How is the learning rate set in the config?, According to the excerpt, what does Ray automatically handle for distributed systems engineers?, source_sentence: RLlib component tree#
The following is the structure of the RLlib component tree, showing under which name you can
access a subcomponent’s own checkpoint within the higher-level checkpoint. At the highest level
is the Algorithm class:
algorithm/
learner_group/
learner/
rl_module/
default_policy/ # <- single-agent case
[module ID 1]/ # <- multi-agent case
[module ID 2]/ # ...
env_runner/
env_to_module_connector/
module_to_env_connector/

Note
The env_runner/ subcomponent currently doesn’t hold a copy of the RLModule
checkpoint because it’s already saved under learner/. The Ray team is working on resolving
this issue, probably through soft-linking to avoid duplicate files and unnecessary disk usage.

Creating instances from a checkpoint with from_checkpoint#
Once you have a checkpoint of either a trained Algorithm or
any of its subcomponents, you can recreate new objects directly
from this checkpoint.
The following are two examples:
sentences: Why does RLlib convert each row into a single-step episode by default?, What is at the highest level of the RLlib component tree?, What is recommended regarding AOF when using storage options that do not support append operations?, source_sentence: Option 2: Manually Create URL (slower to implement, but recommended for production environments)#
The second option is to manually create this URL by pattern-matching your specific use case with one of the following examples.
This is recommended because it provides finer-grained control over which repository branch and commit to use when generating your dependency zip file.
These options prevent consistency issues on Ray Clusters (see the warning above for more info).
To create the URL, pick a URL template below that fits your use case, and fill in all parameters in brackets (e.g. [username], [repository], etc.) with the specific values from your repository.
For instance, suppose your GitHub username is example_user, the repository’s name is example_repository, and the desired commit hash is abcdefg.
If example_repository is public and you want to retrieve the abcdefg commit (which matches the first example use case), the URL would be:
sentences: What can Ray Train and Ray Tune be used together for?, How do you create the URL for Option 2?, Which function can you use to read a CSV file for batch processing in Ray?
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics: cosine_accuracy@1, cosine_accuracy@3, cosine_accuracy@5, cosine_accuracy@10, cosine_precision@1, cosine_precision@3, cosine_precision@5, cosine_precision@10, cosine_recall@1, cosine_recall@3, cosine_recall@5, cosine_recall@10, cosine_ndcg@10, cosine_mrr@10, cosine_map@100
model-index: name: Fine-tune-all-mpnet-base-v2
results: task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 768
type: dim_768
metrics: type: cosine_accuracy@1
value: 0.5874125874125874
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.6818181818181818
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.7954545454545454
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.8863636363636364
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.5874125874125874
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.5180652680652681
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.3944055944055945
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.23199300699300698
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.263986013986014
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.6073717948717948
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.7521853146853147
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.8780594405594405
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.7386606603331115
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.6635614385614379
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.6988731642119342
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 512
type: dim_512
metrics: type: cosine_accuracy@1
value: 0.5734265734265734
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.666083916083916
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.8006993006993007
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.8811188811188811
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.5734265734265734
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.5052447552447552
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.39370629370629373
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.23094405594405593
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.26005244755244755
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.5914918414918414
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.7543706293706294
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.8726689976689977
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.7303335650898982
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.652235958485958
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.689387057080973
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 256
type: dim_256
metrics: type: cosine_accuracy@1
value: 0.5664335664335665
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.666083916083916
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.7797202797202797
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.8583916083916084
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.5664335664335665
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.5011655011655011
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.38636363636363635
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.22534965034965035
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.2577214452214452
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.5893065268065268
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.7354312354312353
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.8487762237762237
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.7167871578299232
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.6432942057942053
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.6823584299690649
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 128
type: dim_128
metrics: type: cosine_accuracy@1
value: 0.5402097902097902
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.6398601398601399
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.743006993006993
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.8304195804195804
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.5402097902097902
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.47960372960372966
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.3678321678321678
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.2181818181818182
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.24519230769230768
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.5623543123543123
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.701048951048951
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.8228438228438228
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.6886328428362513
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.6146582584082584
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.6543671947827556
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 64
type: dim_64
metrics: type: cosine_accuracy@1
value: 0.4353146853146853
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.5332167832167832
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.6311188811188811
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.7622377622377622
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.4353146853146853
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3945221445221445
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.3094405594405594
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.19825174825174827
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.19842657342657344
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.46547202797202797
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.5910547785547785
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.7467948717948718
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.5953015131317417
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.5138784826284825
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.559206100539383
name: Cosine Map@100
verified: no
Likes: 0 | All-time downloads: 11
Created: 2025-06-03 22:24:04