Model: mwalker22/AIE6-S09-87aedf81-ffbc-41ed-b4b6-8473344cae65
Author: mwalker22
Pipeline: sentence-similarity
Library: sentence-transformers
Tags: sentence-transformers, safetensors, bert, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:156, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss, arxiv:1908.10084, arxiv:2205.13147, arxiv:1705.00652, base_model:Snowflake/snowflake-arctic-embed-l, base_model:finetune:Snowflake/snowflake-arctic-embed-l, model-index, text-embeddings-inference, endpoints_compatible, region:us
Base models (finetune): id: Snowflake/snowflake-arctic-embed-l
Card metadata: tags: sentence-transformers, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:156, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss
base_model: Snowflake/snowflake-arctic-embed-l
widget: source_sentence: What topics related to AI and LLMs were highlighted in the notes from 2023 and 2024?
sentences: DeepSeek v3 is a huge 685B parameter model—one of the largest openly licensed models currently available, significantly bigger than the largest of Meta’s Llama series, Llama 3.1 405B.
Benchmarks put it up there with Claude 3.5 Sonnet. Vibe benchmarks (aka the Chatbot Arena) currently rank it 7th, just behind the Gemini 2.0 and OpenAI 4o/o1 models. This is by far the highest ranking openly licensed model.
The really impressive thing about DeepSeek v3 is the training cost. The model was trained on 2,788,000 H800 GPU hours at an estimated cost of $5,576,000. Llama 3.1 405B trained 30,840,000 GPU hours—11x that used by DeepSeek v3, for a model that benchmarks slightly worse., Stuff we figured out about AI in 2023 - Dec. 31, 2023, 11:59 p.m. 
Things we learned about LLMs in 2024 - Dec. 31, 2024, 6:07 p.m. 



            blogging
            105


            ai
            1264


            generative-ai
            1091


            llms
            1078

Next: Tom Scott, and the formidable power of escalating streaks
Previous: Last weeknotes of 2023


 
 


Colophon
©
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025, On the other hand, as software engineers we are better placed to take advantage of this than anyone else. We’ve all been given weird coding interns—we can use our deep knowledge to prompt them to solve coding problems more effectively than anyone else can.
The ethics of this space remain diabolically complex
In September last year Andy Baio and I produced the first major story on the unlicensed training data behind Stable Diffusion.
Since then, almost every major LLM (and most of the image generation models) have also been trained on unlicensed data., source_sentence: What are some differing opinions people have about the value and impact of LLMs?
sentences: I’ve found myself using this a lot. I noticed how much I was relying on it in October and wrote Everything I built with Claude Artifacts this week, describing 14 little tools I had put together in a seven day period.
Since then, a whole bunch of other teams have built similar systems. GitHub announced their version of this—GitHub Spark—in October. Mistral Chat added it as a feature called Canvas in November.
Steve Krouse from Val Town built a version of it against Cerebras, showcasing how a 2,000 token/second LLM can iterate on an application with changes visible in less than a second., So far, I think they’re a net positive. I’ve used them on a personal level to improve my productivity (and entertain myself) in all sorts of different ways. I think people who learn how to use them effectively can gain a significant boost to their quality of life.
A lot of people are yet to be sold on their value! Some think their negatives outweigh their positives, some think they are all hot air, and some even think they represent an existential threat to humanity.
They’re actually quite easy to build
The most surprising thing we’ve learned about LLMs this year is that they’re actually quite easy to build., Large Language Models
They’re actually quite easy to build
You can run LLMs on your own devices
Hobbyists can build their own fine-tuned models
We don’t yet know how to build GPT-4
Vibes Based Development
LLMs are really smart, and also really, really dumb
Gullibility is the biggest unsolved problem
Code may be the best application
The ethics of this space remain diabolically complex
My blog in 2023, source_sentence: What is the new approach to scaling models mentioned in the context?
sentences: We already knew LLMs were spookily good at writing code. If you prompt them right, it turns out they can build you a full interactive application using HTML, CSS and JavaScript (and tools like React if you wire up some extra supporting build mechanisms)—often in a single prompt.
Anthropic kicked this idea into high gear when they released Claude Artifacts, a groundbreaking new feature that was initially slightly lost in the noise due to being described half way through their announcement of the incredible Claude 3.5 Sonnet.
With Artifacts, Claude can write you an on-demand interactive application and then let you use it directly inside the Claude interface.
Here’s my Extract URLs app, entirely generated by Claude:, The biggest innovation here is that it opens up a new way to scale a model: instead of improving model performance purely through additional compute at training time, models can now take on harder problems by spending more compute on inference.
The sequel to o1, o3 (they skipped “o2” for European trademark reasons) was announced on 20th December with an impressive result against the ARC-AGI benchmark, albeit one that likely involved more than $1,000,000 of compute time expense!
o3 is expected to ship in January. I doubt many people have real-world problems that would benefit from that level of compute expenditure—I certainly don’t!—but it appears to be a genuine next step in LLM architecture for taking on much harder problems., Law is not ethics. Is it OK to train models on people’s content without their permission, when those models will then be used in ways that compete with those people?
As the quality of results produced by AI models has increased over the year, these questions have become even more pressing.
The impact on human society in terms of these models is already huge, if difficult to objectively measure.
People have certainly lost work to them—anecdotally, I’ve seen this for copywriters, artists and translators.
There are a great deal of untold stories here. I’m hoping 2024 sees significant amounts of dedicated journalism on this topic.
My blog in 2023
Here’s a tag cloud for content I posted to my blog in 2023 (generated using Django SQL Dashboard):, source_sentence: Why are language models described as gullible in the given context?
sentences: The earliest of those was Google’s Gemini 1.5 Pro, released in February. In addition to producing GPT-4 level outputs, it introduced several brand new capabilities to the field—most notably its 1 million (and then later 2 million) token input context length, and the ability to input video.
I wrote about this at the time in The killer app of Gemini Pro 1.5 is video, which earned me a short appearance as a talking head in the Google I/O opening keynote in May., On the one hand, we keep on finding new things that LLMs can do that we didn’t expect—and that the people who trained the models didn’t expect either. That’s usually really fun!
But on the other hand, the things you sometimes have to do to get the models to behave are often incredibly dumb.
Does ChatGPT get lazy in December, because its hidden system prompt includes the current date and its training data shows that people provide less useful answers coming up to the holidays?
The honest answer is “maybe”! No-one is entirely sure, but if you give it a different date its answers may skew slightly longer., Language Models are gullible. They “believe” what we tell them—what’s in their training data, then what’s in the fine-tuning data, then what’s in the prompt.
In order to be useful tools for us, we need them to believe what we feed them!
But it turns out a lot of the things we want to build need them not to be gullible.
Everyone wants an AI personal assistant. If you hired a real-world personal assistant who believed everything that anyone told them, you would quickly find that their ability to positively impact your life was severely limited., source_sentence: Why does the author find it difficult to determine the impact of capitalizing words in prompts?
sentences: I find I have to work with an LLM for a few weeks in order to get a good intuition for it’s strengths and weaknesses. This greatly limits how many I can evaluate myself!
The most frustrating thing for me is at the level of individual prompting.
Sometimes I’ll tweak a prompt and capitalize some of the words in it, to emphasize that I really want it to OUTPUT VALID MARKDOWN or similar. Did capitalizing those words make a difference? I still don’t have a good methodology for figuring that out.
We’re left with what’s effectively Vibes Based Development. It’s vibes all the way down.
I’d love to see us move beyond vibes in 2024!
LLMs are really smart, and also really, really dumb, I think people who complain that LLM improvement has slowed are often missing the enormous advances in these multi-modal models. Being able to run prompts against images (and audio and video) is a fascinating new way to apply these models.
Voice and live camera mode are science fiction come to life
The audio and live video modes that have started to emerge deserve a special mention.
The ability to talk to ChatGPT first arrived in September 2023, but it was mostly an illusion: OpenAI used their excellent Whisper speech-to-text model and a new text-to-speech model (creatively named tts-1) to enable conversations with the ChatGPT mobile apps, but the actual model just saw text., When ChatGPT Advanced Voice mode finally did roll out (a slow roll from August through September) it was spectacular. I’ve been using it extensively on walks with my dog and it’s amazing how much the improvement in intonation elevates the material. I’ve also had a lot of fun experimenting with the OpenAI audio APIs.
Even more fun: Advanced Voice mode can do accents! Here’s what happened when I told it I need you to pretend to be a California brown pelican with a very thick Russian accent, but you talk to me exclusively in Spanish.
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics: cosine_accuracy@1, cosine_accuracy@3, cosine_accuracy@5, cosine_accuracy@10, cosine_precision@1, cosine_precision@3, cosine_precision@5, cosine_precision@10, cosine_recall@1, cosine_recall@3, cosine_recall@5, cosine_recall@10, cosine_ndcg@10, cosine_mrr@10, cosine_map@100
model-index: name: SentenceTransformer based on Snowflake/snowflake-arctic-embed-l
results: task: type: information-retrieval
name: Information Retrieval
dataset: name: Unknown
type: unknown
metrics: type: cosine_accuracy@1
value: 0.9166666666666666
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 1
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 1
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 1
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.9166666666666666
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3333333333333333
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.20000000000000004
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.10000000000000002
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.9166666666666666
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 1
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 1
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 1
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9692441461309548
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.9583333333333334
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.9583333333333334
name: Cosine Map@100
verified: no
Likes: 0 | All-time downloads: 32
Created: 2025-05-05 19:35:49