Model: dabraldeepti25/legal-ft-v0-updated
Author: dabraldeepti25
Pipeline: sentence-similarity
Library: sentence-transformers
Tags: sentence-transformers, safetensors, bert, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:156, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss, arxiv:1908.10084, arxiv:2205.13147, arxiv:1705.00652, base_model:Snowflake/snowflake-arctic-embed-l, base_model:finetune:Snowflake/snowflake-arctic-embed-l, model-index, text-embeddings-inference, endpoints_compatible, region:us
Base models (finetune): id: Snowflake/snowflake-arctic-embed-l
Card metadata: tags: sentence-transformers, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:156, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss
base_model: Snowflake/snowflake-arctic-embed-l
widget: source_sentence: What significant change occurred in the AI landscape regarding models surpassing GPT-4 in the past twelve months?
sentences: Except... you can run generated code to see if it’s correct. And with patterns like ChatGPT Code Interpreter the LLM can execute the code itself, process the error message, then rewrite it and keep trying until it works!
So hallucination is a much lesser problem for code generation than for anything else. If only we had the equivalent of Code Interpreter for fact-checking natural language!
How should we feel about this as software engineers?
On the one hand, this feels like a threat: who needs a programmer if ChatGPT can write code for you?, The GPT-4 barrier was comprehensively broken
In my December 2023 review I wrote about how We don’t yet know how to build GPT-4—OpenAI’s best model was almost a year old at that point, yet no other AI lab had produced anything better. What did OpenAI know that the rest of us didn’t?
I’m relieved that this has changed completely in the past twelve months. 18 organizations now have models on the Chatbot Arena Leaderboard that rank higher than the original GPT-4 from March 2023 (GPT-4-0314 on the board)—70 models in total., If you think about what they do, this isn’t such a big surprise. The grammar rules of programming languages like Python and JavaScript are massively less complicated than the grammar of Chinese, Spanish or English.
It’s still astonishing to me how effective they are though.
One of the great weaknesses of LLMs is their tendency to hallucinate—to imagine things that don’t correspond to reality. You would expect this to be a particularly bad problem for code—if an LLM hallucinates a method that doesn’t exist, the code should be useless., source_sentence: How does Claude enable users to interact with applications created through its interface?
sentences: OpenAI made GPT-4o free for all users in May, and Claude 3.5 Sonnet was freely available from its launch in June. This was a momentus change, because for the previous year free users had mostly been restricted to GPT-3.5 level models, meaning new users got a very inaccurate mental model of what a capable LLM could actually do.
That era appears to have ended, likely permanently, with OpenAI’s launch of ChatGPT Pro. This $200/month subscription service is the only way to access their most capable model, o1 Pro.
Since the trick behind the o1 series (and the future models it will undoubtedly inspire) is to expend more compute time to get better results, I don’t think those days of free access to the best available models are likely to return., We already knew LLMs were spookily good at writing code. If you prompt them right, it turns out they can build you a full interactive application using HTML, CSS and JavaScript (and tools like React if you wire up some extra supporting build mechanisms)—often in a single prompt.
Anthropic kicked this idea into high gear when they released Claude Artifacts, a groundbreaking new feature that was initially slightly lost in the noise due to being described half way through their announcement of the incredible Claude 3.5 Sonnet.
With Artifacts, Claude can write you an on-demand interactive application and then let you use it directly inside the Claude interface.
Here’s my Extract URLs app, entirely generated by Claude:, Industry’s Tardy Response to the AI Prompt Injection Vulnerability on RedMonk Conversations


Posted 31st December 2023 at 11:59 pm · Follow me on Mastodon or Twitter or subscribe to my newsletter


More recent articles

LLM 0.22, the annotated release notes - 17th February 2025
Run LLMs on macOS using llm-mlx and Apple's MLX framework - 15th February 2025
URL-addressable Pyodide Python environments - 13th February 2025


 


This is Stuff we figured out about AI in 2023 by Simon Willison, posted on 31st December 2023.

Part of series LLMs annual review

Stuff we figured out about AI in 2023 - Dec. 31, 2023, 11:59 p.m. 
Things we learned about LLMs in 2024 - Dec. 31, 2024, 6:07 p.m. 



            blogging
            69, source_sentence: What incident involving Google Search is mentioned in the context, and what was the nature of the misinformation?
sentences: My personal laptop is a 64GB M2 MacBook Pro from 2023. It’s a powerful machine, but it’s also nearly two years old now—and crucially it’s the same laptop I’ve been using ever since I first ran an LLM on my computer back in March 2023 (see Large language models are having their Stable Diffusion moment).
That same laptop that could just about run a GPT-3-class model in March last year has now run multiple GPT-4 class models! Some of my notes on that:, Terminology aside, I remain skeptical as to their utility based, once again, on the challenge of gullibility. LLMs believe anything you tell them. Any systems that attempts to make meaningful decisions on your behalf will run into the same roadblock: how good is a travel agent, or a digital assistant, or even a research tool if it can’t distinguish truth from fiction?
Just the other day Google Search was caught serving up an entirely fake description of the non-existant movie “Encanto 2”. It turned out to be summarizing an imagined movie listing from a fan fiction wiki., On the other hand, as software engineers we are better placed to take advantage of this than anyone else. We’ve all been given weird coding interns—we can use our deep knowledge to prompt them to solve coding problems more effectively than anyone else can.
The ethics of this space remain diabolically complex
In September last year Andy Baio and I produced the first major story on the unlicensed training data behind Stable Diffusion.
Since then, almost every major LLM (and most of the image generation models) have also been trained on unlicensed data., source_sentence: What are the limitations of Apple's LLM features compared to frontier LLMs, according to the context?
sentences: DeepSeek v3 is a huge 685B parameter model—one of the largest openly licensed models currently available, significantly bigger than the largest of Meta’s Llama series, Llama 3.1 405B.
Benchmarks put it up there with Claude 3.5 Sonnet. Vibe benchmarks (aka the Chatbot Arena) currently rank it 7th, just behind the Gemini 2.0 and OpenAI 4o/o1 models. This is by far the highest ranking openly licensed model.
The really impressive thing about DeepSeek v3 is the training cost. The model was trained on 2,788,000 H800 GPU hours at an estimated cost of $5,576,000. Llama 3.1 405B trained 30,840,000 GPU hours—11x that used by DeepSeek v3, for a model that benchmarks slightly worse., An interesting point of comparison here could be the way railways rolled out around the world in the 1800s. Constructing these required enormous investments and had a massive environmental impact, and many of the lines that were built turned out to be unnecessary—sometimes multiple lines from different companies serving the exact same routes!
The resulting bubbles contributed to several financial crashes, see Wikipedia for Panic of 1873, Panic of 1893, Panic of 1901 and the UK’s Railway Mania. They left us with a lot of useful infrastructure and a great deal of bankruptcies and environmental damage.
The year of slop, Now that those features are rolling out they’re pretty weak. As an LLM power-user I know what these models are capable of, and Apple’s LLM features offer a pale imitation of what a frontier LLM can do. Instead we’re getting notification summaries that misrepresent news headlines and writing assistant tools that I’ve not found useful at all. Genmoji are kind of fun though.
The rise of inference-scaling “reasoning” models
The most interesting development in the final quarter of 2024 was the introduction of a new shape of LLM, exemplified by OpenAI’s o1 models—initially released as o1-preview and o1-mini on September 12th., source_sentence: What new feature was introduced in ChatGPT's voice mode in December?
sentences: The most recent twist, again from December (December was a lot) is live video. ChatGPT voice mode now provides the option to share your camera feed with the model and talk about what you can see in real time. Google Gemini have a preview of the same feature, which they managed to ship the day before ChatGPT did., The two main categories I see are people who think AI agents are obviously things that go and act on your behalf—the travel agent model—and people who think in terms of LLMs that have been given access to tools which they can run in a loop as part of solving a problem. The term “autonomy” is often thrown into the mix too, again without including a clear definition.
(I also collected 211 definitions on Twitter a few months ago—here they are in Datasette Lite—and had gemini-exp-1206 attempt to summarize them.)
Whatever the term may mean, agents still have that feeling of perpetually “coming soon”., The GPT-4 barrier was comprehensively broken
In my December 2023 review I wrote about how We don’t yet know how to build GPT-4—OpenAI’s best model was almost a year old at that point, yet no other AI lab had produced anything better. What did OpenAI know that the rest of us didn’t?
I’m relieved that this has changed completely in the past twelve months. 18 organizations now have models on the Chatbot Arena Leaderboard that rank higher than the original GPT-4 from March 2023 (GPT-4-0314 on the board)—70 models in total.
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics: cosine_accuracy@1, cosine_accuracy@3, cosine_accuracy@5, cosine_accuracy@10, cosine_precision@1, cosine_precision@3, cosine_precision@5, cosine_precision@10, cosine_recall@1, cosine_recall@3, cosine_recall@5, cosine_recall@10, cosine_ndcg@10, cosine_mrr@10, cosine_map@100
model-index: name: SentenceTransformer based on Snowflake/snowflake-arctic-embed-l
results: task: type: information-retrieval
name: Information Retrieval
dataset: name: Unknown
type: unknown
metrics: type: cosine_accuracy@1
value: 0.9166666666666666
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 1
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 1
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 1
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.9166666666666666
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3333333333333333
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.20000000000000004
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.10000000000000002
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.9166666666666666
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 1
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 1
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 1
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9692441461309548
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.9583333333333334
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.9583333333333334
name: Cosine Map@100
verified: no
Likes: 0 | All-time downloads: 51
Created: 2025-02-18 22:13:53