Netta1994/setfit_baai_gpt-4o_cot-few_shot_remove_final_evaluation_e1_one_big_model_1727080822.0

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: BAAI/bge-base-en-v1.5
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 512 tokens
• Number of Classes: 2 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
0	"The answer incorporates several elements not mentioned in the provided document, specifically the references to a virtual reality training technique and its impact on player decision-making. These aspects are not mentioned in the document, rendering the information inaccurate.\n\nIn the actual document, the offensive outburst of the Nuggets is attributed to coach Brian Shaw's strategy of encouraging players to take the first available shot in the rhythm of the offense and push the ball after makes and misses. The comfort and effectiveness in these strategies coming together are cited as reasons for the increased scoring.\n\nTherefore, the provided answer is flawed due to the inclusionof fabricated details.\n\nThe final evaluation:" 'The answer provided contains several inaccuracies and fabrications that do not align with the content of the document.\n\n1. Film Under-Exposure Statement: The answer erroneously states that "film under-exposes better than a digital sensor," whereas the document clearly mentions that "film over-exposes better than a digital sensor."\n\n2. Color Compression Errors: The answer claims film compresses exposure range into the "bottom end" and colors saturate to black, but the document specifies it compresses into the "top end" and colors desaturate to white.\n\n3. Sensor Details: The answer inaccurately mentions that digital sensors capture all three colors at each point when in reality it is stated that "Film also captures all three colors at every point. Digital sensors (all but Fovian, anyway) capture only one color at each point and then interpolate between them."\n\n4. Megapixel Comparison: The claim that the author finds "5MP digital sensors of today to be about comparable to high-end, professional film" is incorrect. The document actually compares "10MP digital sensors of today" to common, non-professional film for resolution.\n\nGiven these significant discrepancies and inaccuracies, the answer provided is unreliable and does not accurately reflect the document's content.\n\nThe final evaluation:' 'The provided answer addresses an entirely different topic—providing details about fighters and outcomes from a mixed martial arts event rather than discussing the main conflict in the third book of the Arcana Chronicles by Kresley Cole. The answer did not address the question at all. \n\nFinal evaluation:'
1	"The answer provided addresses the key elements that align with the best practices outlined in the document:\n\n1. Getting to Know the Client: The answer mentions understanding the client's needs, wants, and goals before starting the web design process, which is directly echoed in the document.\n\n2. Signing a Contract: The answer highlights the importance of having a detailed contract that outlines the scope of the project, costs, and how future revisions will be managed. This ensures that there are clear parameters and a point of reference if excessive requests arise.\n\n3. Honesty and Diplomacy: The answer advises showcasing a sense of honesty and diplomacy, particularly when extra charges are necessary or when certain requests are unfeasible. This aligns with the document's advice on effective communication and managing client expectations diplomatically.\n\nOverall, the answer aligns well with the recommendations provided in the document.\n\nThe final evaluation:" "The answer provided is accurate and aligns well with the content of the document. The document discusses the importance of drawing on an author's own emotional experiences, particularly pain and emotion, to create genuine and relatable characters. This approach helps forge a connection between the reader and the characters.\n\nFinal evaluation:" 'The answer is directly substantiated by the document. It clearly mentions that Mauro Rubin, the CEO of JoinPad, was present at the event at Talent Garden Calabiana, Milan. The answer is concise and provides the exact information asked in the question without any extraneous details. \n\nFinal evaluation:'

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_gpt-4o_cot-few_shot_remove_final_evaluation_e1_one_big_model_1727080822.0")
# Run inference
preds = model("The answer provided accurately states that Allan Cox's First Class Delivery was launched on a H128-10W for his Level 1 certification flight. This information is directly retrieved from the document.

The final evaluation:")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	12	75.0147	301

Label	Training Sample Count
0	199
1	209

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (1, 1)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (2e-05, 2e-05)
• head_learning_rate: 2e-05
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0010	1	0.2249	-
0.0490	50	0.2456	-
0.0980	100	0.1748	-
0.1471	150	0.0861	-
0.1961	200	0.051	-
0.2451	250	0.0613	-
0.2941	300	0.0325	-
0.3431	350	0.0128	-
0.3922	400	0.0075	-
0.4412	450	0.007	-
0.4902	500	0.004	-
0.5392	550	0.0027	-
0.5882	600	0.0023	-
0.6373	650	0.0019	-
0.6863	700	0.0018	-
0.7353	750	0.0017	-
0.7843	800	0.0017	-
0.8333	850	0.0016	-
0.8824	900	0.0016	-
0.9314	950	0.0015	-
0.9804	1000	0.0014	-

Framework Versions

• Python: 3.10.14
• SetFit: 1.1.0
• Sentence Transformers: 3.1.1
• Transformers: 4.44.0
• PyTorch: 2.4.0+cu121
• Datasets: 3.0.0
• Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}