Netta1994/setfit_baai_newrelic_gpt-4o_cot-instructions_remove_final_evaluation_e1_one_out_17270

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: BAAI/bge-base-en-v1.5
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 512 tokens
• Number of Classes: 2 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
1	'Reasoning:\n1. Context Grounding: The provided answer accurately reflects the information in the document that the Partner Dashboard, including the calendar, cannot currently be accessed via mobile devices and requires supported desktop browsers. This aligns perfectly with the statement in the document.\n2. Relevance: The response is directly related to the question about accessing the calendar on mobile devices.\n3. Conciseness: The answer is clear and straightforward, avoiding unnecessary details while keeping focused on the main point.\n4. Correct and Detailed Instructions: The answer provides correct instructions, including the suggestion to vote for the feature as a form of showing interest, which is detailed in the document.\n\nFinal Result:' 'Reasoning:\n\n1. Context Grounding: The provided answer accurately reflects the information from the document regarding the 2002 Chrysler Concorde LX for sale in Orlando, FL. The price and mileage data are directly taken from the document.\n2. Relevance: The answer is specifically focused on the price and mileage of the 2002 Chrysler Concorde LX in Orlando, FL, which is precisely what the question asked.\n3. Conciseness: The answer is concise and to the point, providing only the necessary information without any additional, unnecessary details.\n\nFinal Result:' 'Reasoning:\n1. Context Grounding: The answer needs to be verified against the information in the document. The document lists "DeleteQuery" as an event name but it should be confirmed to ensure accuracy.\n2. Relevance: The answer specifically relates to the value of the key name when the event name is DeleteQuery.\n3. Conciseness: The answer should provide only the necessary value without additional, unrelated information.\n4. Specificity: The answer should directly specify the value corresponding to for the event name DeleteQuery.\n5. Key/Value/Event Accuracy: The provided answer needs to match the correct value in the document.\n\nBased on the document:\n- Event name(s) is DeleteQuery\n- Anthonyview key name is \n- Value is queryName\n\nThe provided answer "queryName" is correct, as it matches the document precisely.\n\nFinal Result:'
0	'**Reasoning:\n\n1. Context Grounding: The provided document does not mention "Father Josh Carrier" but clearly speaks about "Father Joseph Carrier, C.S.C." and details his roles. Thus, the argument surrounding "Father Josh Carrier" is not properly grounded in the provided context. Additionally, the document does not support the claim that he was a football coach.\n \n2. Relevance: While the answer attempts to address the professorship held, it inaccurately names the individual (Josh instead of Joseph) and adds irrelevant and incorrect information about him being a football coach.\n \n3. Conciseness: The answer introduces unnecessary details by incorrectly stating he was a football coach, which deviates from the core question.\n\nFinal Result: **' "Reasoning:\n1. Context Grounding: The answer does not accurately reflect the document's context. The document explains how to filter posts in the blog dashboard for site administrators, not how to add filters for visitors to search blog posts.\n2. Relevance: The answer does not address the visitor experience or how to set up filters for visitors. Instead, it details steps for internal filtering through the dashboard.\n3. Conciseness: While the answer is concise, it is not relevant to the question asked.\n4. Correctness and Detail: The instructions provided do not pertain to setting up filters for visitor searches on the blog, making them incorrect for the question posed.\n\nFinal Result:" "Reasoning:\n1. Context Grounding: The answer begins with the process of cutting fishing line, which is aligned with the document, and it also mentions the need for extra line, reflecting the document details. The use of overhand or square knots and stringing from the unknotted end follows the document's instructions. However, the mention of ancient beaders and the full moon is not grounded in the provided document, which is an unnecessary and unrelated addition.\n2. Relevance: The answer predominantly covers the relevant steps for beading, such as cutting the fishing line, stringing beads, and tying knots. However, the historical reference about beaders and the full moon does not address how to bead and detracts from the main task.\n3. Conciseness: The answer is relatively concise but could exclude the historical context that does not contribute to the step-by-step beading process, making it a bit cluttered.\n\nFinal Result:"

Evaluation

Metrics

Label	Accuracy
all	1.0

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_cot-instructions_remove_final_evaluation_e1_one_out_17270")
# Run inference
preds = model("Reasoning:
1. Context Grounding: The response draws from the documents providing relevant sources such as the organization's website, job ads, and newsletter link.
2. Relevance: The answer is directly related to the question about understanding the organization's products, challenges, and future.
3. Conciseness: The answer is clear and to the point.
4. Does not attempt to respond when the document lacks information: It addresses the question appropriately with the available information.
5. Specificity: The answer is specific and provides concrete steps to follow.
6. Relevant tips: The answer includes actionable steps like visiting the website, viewing job ads, and signing up for a newsletter, which are relevant.

The answer precisely matches all the criteria set for evaluation.

Final Result:")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	32	104.4858	245

Label	Training Sample Count
0	381
1	393

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (1, 1)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (2e-05, 2e-05)
• head_learning_rate: 2e-05
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0005	1	0.2622	-
0.0258	50	0.2676	-
0.0517	100	0.2524	-
0.0775	150	0.2505	-
0.1034	200	0.253	-
0.1292	250	0.2458	-
0.1550	300	0.2191	-
0.1809	350	0.1842	-
0.2067	400	0.1665	-
0.2326	450	0.1262	-
0.2584	500	0.093	-
0.2842	550	0.0614	-
0.3101	600	0.0524	-
0.3359	650	0.0346	-
0.3618	700	0.0412	-
0.3876	750	0.0246	-
0.4134	800	0.0183	-
0.4393	850	0.0165	-
0.4651	900	0.0193	-
0.4910	950	0.0134	-
0.5168	1000	0.0044	-
0.5426	1050	0.0097	-
0.5685	1100	0.0085	-
0.5943	1150	0.0088	-
0.6202	1200	0.0079	-
0.6460	1250	0.0042	-
0.6718	1300	0.003	-
0.6977	1350	0.0038	-
0.7235	1400	0.0072	-
0.7494	1450	0.0017	-
0.7752	1500	0.0024	-
0.8010	1550	0.0019	-
0.8269	1600	0.0015	-
0.8527	1650	0.0015	-
0.8786	1700	0.0014	-
0.9044	1750	0.0014	-
0.9302	1800	0.0014	-
0.9561	1850	0.0014	-
0.9819	1900	0.0013	-

Framework Versions

• Python: 3.10.14
• SetFit: 1.1.0
• Sentence Transformers: 3.1.1
• Transformers: 4.44.0
• PyTorch: 2.4.0+cu121
• Datasets: 3.0.0
• Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}