Model: mwalker22/AIE6-S09-e9fadcc8-88f5-4b9a-bd89-a2deb5ae3af7
Author: mwalker22
Pipeline: sentence-similarity
Library: sentence-transformers
Tags: sentence-transformers, safetensors, bert, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:157, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss, arxiv:1908.10084, arxiv:2205.13147, arxiv:1705.00652, base_model:Snowflake/snowflake-arctic-embed-l, base_model:finetune:Snowflake/snowflake-arctic-embed-l, model-index, text-embeddings-inference, endpoints_compatible, region:us
Base models (finetune): id: Snowflake/snowflake-arctic-embed-l
Card metadata: tags: sentence-transformers, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:157, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss
base_model: Snowflake/snowflake-arctic-embed-l
widget: source_sentence: What factors have contributed to the improved efficiency and performance of these models over the past year?
sentences: The May 13th announcement of GPT-4o included a demo of a brand new voice mode, where the true multi-modal GPT-4o (the o is for “omni”) model could accept audio input and output incredibly realistic sounding speech without needing separate TTS or STT models.
The demo also sounded conspicuously similar to Scarlett Johansson... and after she complained the voice from the demo, Skye, never made it to a production product.
The delay in releasing the new voice mode after the initial demo caused quite a lot of confusion. I wrote about that in ChatGPT in “4o” mode is not running the new features yet., This remains astonishing to me. I thought a model with the capabilities and output quality of GPT-4 needed a datacenter class server with one or more $40,000+ GPUs.
These models take up enough of my 64GB of RAM that I don’t run them often—they don’t leave much room for anything else.
The fact that they run at all is a testament to the incredible training and inference performance gains that we’ve figured out over the past year. It turns out there was a lot of low-hanging fruit to be harvested in terms of model efficiency. I expect there’s still more to come., Your browser does not support the audio element.

OpenAI aren’t the only group with a multi-modal audio model. Google’s Gemini also accepts audio input, and the Google Gemini apps can speak in a similar way to ChatGPT now. Amazon also pre-announced voice mode for Amazon Nova, but that’s meant to roll out in Q1 of 2025.
Google’s NotebookLM, released in September, took audio output to a new level by producing spookily realistic conversations between two “podcast hosts” about anything you fed into their tool. They later added custom instructions, so naturally I turned them into pelicans:


Your browser does not support the audio element., source_sentence: Why is gullibility considered the biggest unsolved problem in the context of large language models?
sentences: Except... you can run generated code to see if it’s correct. And with patterns like ChatGPT Code Interpreter the LLM can execute the code itself, process the error message, then rewrite it and keep trying until it works!
So hallucination is a much lesser problem for code generation than for anything else. If only we had the equivalent of Code Interpreter for fact-checking natural language!
How should we feel about this as software engineers?
On the one hand, this feels like a threat: who needs a programmer if ChatGPT can write code for you?, Large Language Models
They’re actually quite easy to build
You can run LLMs on your own devices
Hobbyists can build their own fine-tuned models
We don’t yet know how to build GPT-4
Vibes Based Development
LLMs are really smart, and also really, really dumb
Gullibility is the biggest unsolved problem
Code may be the best application
The ethics of this space remain diabolically complex
My blog in 2023, Prompt injection is a natural consequence of this gulibility. I’ve seen precious little progress on tackling that problem in 2024, and we’ve been talking about it since September 2022.
I’m beginning to see the most popular idea of “agents” as dependent on AGI itself. A model that’s robust against gulliblity is a very tall order indeed.
Evals really matter
Anthropic’s Amanda Askell (responsible for much of the work behind Claude’s Character):, source_sentence: What challenge does the author highlight about evaluating the performance of LLMs?
sentences: OpenAI are not the only game in town here. Google released their first entrant in the category, gemini-2.0-flash-thinking-exp, on December 19th.
Alibaba’s Qwen team released their QwQ model on November 28th—under an Apache 2.0 license, and that one I could run on my own machine. They followed that up with a vision reasoning model called QvQ on December 24th, which I also ran locally.
DeepSeek made their DeepSeek-R1-Lite-Preview model available to try out through their chat interface on November 20th.
To understand more about inference scaling I recommend Is AI progress slowing down? by Arvind Narayanan and Sayash Kapoor., Still, I’m surprised that no-one has beaten the now almost year old GPT-4 by now. OpenAI clearly have some substantial tricks that they haven’t shared yet.
Vibes Based Development
As a computer scientist and software engineer, LLMs are infuriating.
Even the openly licensed ones are still the world’s most convoluted black boxes. We continue to have very little idea what they can do, how exactly they work and how best to control them.
I’m used to programming where the computer does exactly what I tell it to do. Prompting an LLM is decidedly not that!
The worst part is the challenge of evaluating them.
There are plenty of benchmarks, but no benchmark is going to tell you if an LLM actually “feels” right when you try it for a given task., Now that those features are rolling out they’re pretty weak. As an LLM power-user I know what these models are capable of, and Apple’s LLM features offer a pale imitation of what a frontier LLM can do. Instead we’re getting notification summaries that misrepresent news headlines and writing assistant tools that I’ve not found useful at all. Genmoji are kind of fun though.
The rise of inference-scaling “reasoning” models
The most interesting development in the final quarter of 2024 was the introduction of a new shape of LLM, exemplified by OpenAI’s o1 models—initially released as o1-preview and o1-mini on September 12th., source_sentence: What challenge related to prompt injection has seen little progress since September 2022?
sentences: So far, I think they’re a net positive. I’ve used them on a personal level to improve my productivity (and entertain myself) in all sorts of different ways. I think people who learn how to use them effectively can gain a significant boost to their quality of life.
A lot of people are yet to be sold on their value! Some think their negatives outweigh their positives, some think they are all hot air, and some even think they represent an existential threat to humanity.
They’re actually quite easy to build
The most surprising thing we’ve learned about LLMs this year is that they’re actually quite easy to build., In 2024, almost every significant model vendor released multi-modal models. We saw the Claude 3 series from Anthropic in March, Gemini 1.5 Pro in April (images, audio and video), then September brought Qwen2-VL and Mistral’s Pixtral 12B and Meta’s Llama 3.2 11B and 90B vision models. We got audio input and output from OpenAI in October, then November saw SmolVLM from Hugging Face and December saw image and video models from Amazon Nova.
In October I upgraded my LLM CLI tool to support multi-modal models via attachments. It now has plugins for a whole collection of different vision models., Prompt injection is a natural consequence of this gulibility. I’ve seen precious little progress on tackling that problem in 2024, and we’ve been talking about it since September 2022.
I’m beginning to see the most popular idea of “agents” as dependent on AGI itself. A model that’s robust against gulliblity is a very tall order indeed.
Evals really matter
Anthropic’s Amanda Askell (responsible for much of the work behind Claude’s Character):, source_sentence: What is the main reason given for the lack of AI agents running in production despite many prototypes?
sentences: A lot of people are excited about AI agents—an infuriatingly vague term that seems to be converging on “AI systems that can go away and act on your behalf”. We’ve been talking about them all year, but I’ve seen few if any examples of them running in production, despite lots of exciting prototypes.
I think this is because of gullibility.
Can we solve this? Honestly, I’m beginning to suspect that you can’t fully solve gullibility without achieving AGI. So it may be quite a while before those agent dreams can really start to come true!
Code may be the best application
Over the course of the year, it’s become increasingly clear that writing code is one of the things LLMs are most capable of., The year of slop
Synthetic training data works great
LLMs somehow got even harder to use
Knowledge is incredibly unevenly distributed
LLMs need better criticism
Everything tagged “llms” on my blog in 2024, Longer inputs dramatically increase the scope of problems that can be solved with an LLM: you can now throw in an entire book and ask questions about its contents, but more importantly you can feed in a lot of example code to help the model correctly solve a coding problem. LLM use-cases that involve long inputs are far more interesting to me than short prompts that rely purely on the information already baked into the model weights. Many of my tools were built using this pattern.
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics: cosine_accuracy@1, cosine_accuracy@3, cosine_accuracy@5, cosine_accuracy@10, cosine_precision@1, cosine_precision@3, cosine_precision@5, cosine_precision@10, cosine_recall@1, cosine_recall@3, cosine_recall@5, cosine_recall@10, cosine_ndcg@10, cosine_mrr@10, cosine_map@100
model-index: name: SentenceTransformer based on Snowflake/snowflake-arctic-embed-l
results: task: type: information-retrieval
name: Information Retrieval
dataset: name: Unknown
type: unknown
metrics: type: cosine_accuracy@1
value: 0.875
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 1
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 1
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 1
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.875
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3333333333333333
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.20000000000000004
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.10000000000000002
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.875
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 1
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 1
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 1
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9538662191964322
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.9375
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.9375
name: Cosine Map@100
verified: no
Likes: 0 | All-time downloads: 35
Created: 2025-05-03 07:37:27