leyendo estos comentarios porque O1 no se puede ver en el direct chat?: ""

Dom — ayer a las 16:18
that still remains to be proven tbh 
thus far it's far away from being better in every way from base model
wild — ayer a las 16:18
o1 mini if it's just based on 4o mini, is far far better than 4o mini, generally, it seems
Dom — ayer a las 16:18
this just looks like a risky move that might not ultimately result in much at all, especially at that price with extra tokens on top
moozooh — ayer a las 16:19
i'm not sure why you think this new dimension should be expected to make a model better in every way. not every task requires deeper reasoning
Dom — ayer a las 16:21
because the pricing doesn't make sense and they are in the business of making general purpose models. And truthfully you can fine-tune any model to be better at certain things while degrading the others, with no cost increase :catgrin: 
moozooh — ayer a las 16:22
yeah, i'm pretty sure that if fine-tuning alone could let regular models do what the o1 family does on math evals, the other labs would've done it months ago
Dom — ayer a las 16:23
ehm look at qwen2.5 72b math eval
and then compare it to o1-preview
and that one wasn't even finetuned
and is small kinda, no?
bottom line is openai have much more resources than almost anyone else, so of course they gonna make it 'work'. But if that's an optimal solution this is a different question entirely 
moozooh — ayer a las 16:26
last i checked, 94.8 on MATH is a bit bigger than 83.1, and their size is comparable at least
Dom — ayer a las 16:27
Imagen
moozooh — ayer a las 16:27
Imagen
Dom — ayer a las 16:27
they reference that directly against their other models, so it can't exactly be wrong score
https://github.com/openai/simple-evals?tab=readme-ov-file#benchmark-results\
GitHub
GitHub - openai/simple-evals
Contribute to openai/simple-evals development by creating an account on GitHub.
GitHub - openai/simple-evals
Dom — ayer a las 16:28
that looks kinda misleading and it's straight from their marketing blogpost
moozooh — ayer a las 16:28
did you actually cut out the bottom of that table?
Imagen
Dom — ayer a las 16:29
I did and then I linked the rest
cause we talking about o1-preview
moozooh — ayer a las 16:29
are we?
Dom — ayer a las 16:29
?
⁠general⁠
moozooh — ayer a las 16:30
i was talking about the o1 family because the improvement applies to all of them, and the preview o1 is exactly that: a preview
it's clear that the full thing scales it up further
Dom — ayer a las 16:31
well I can only assess what is already there. People should do the same tbh given how gemini ultra 1.0 turned out
wild — ayer a las 16:32
idk o1 mini seemed actually impressive to me, compared to 1.0 ultra where you could tell that it was bad from the start if you used it 
Dom — ayer a las 16:33
I was referring more to the part of trusting some metrics of a model that does not exist
moozooh — ayer a las 16:33
ok, o1 mini scores 90 on MATH, is that also fine-tuning?
Dom — ayer a las 16:34
even if it's released and it all checks out
by that time, we might have different models and different perspective
moozooh — ayer a las 16:36
it's a bit hard to understand how to argue against purely speculative arguments while actual evals are written off
o1 sets new records on a bunch of metrics: unproven, could be achieved via fine-tuning
like... okay
Bob — ayer a las 16:37
Anyone knows what happens when an image is uploaded to the website? Is it automatically downscaled by them before they send it to the model's server? If yes to what resolution? And it usually uses the lower quality modes that models offer?
Dom — ayer a las 16:38
lol what. I only "write off" evals for models that are not released yet. I think I was clear enough
and anyway, you seem fixated on the things that you like and fail to see the bigger picture kind of ⁠general⁠
moozooh — ayer a las 16:41
that's so cute
Dom — ayer a las 16:41
you are weird 🙂
moozooh — ayer a las 16:41
you just keep making speculative arguments and ignoring eval data
have you actually looked at the table you linked earlier?
Dom — ayer a las 16:43
?
Dom — ayer a las 16:43
I'm not ignoring anything, I'm literally linking it. But if you are considering marketing post evals on stuff not many other models were even tested on at all, I'm not too surprised tbh
moozooh — ayer a las 16:45
if you think openai lied about o1-wip evals, look at the math and gpqa results for o1-preview and tell me that is fine-tuning
i need to see the limit to which you can stretch this sorry argument
Dom — ayer a las 16:46
I'm struggling to find any good reasons to continue arguing with you tbh 
you just seem delusional
moozooh — ayer a las 16:46
lmao
Dom — ayer a las 16:48
if you think evals for o1-preview for that price are super cool, look at evals GPT4 og versus gpt4o. And as for math I already mentioned a certain sub 100b model. Gonna leave you with that. 😉
moozooh — ayer a las 16:48
what a wonderful example of moving the goalpost
""