MurphyA/hacker-news huggingface dataset:
by saleban1031 | 2025-05-29 04:07:42+00:00
While building a RAG system, I got frustrated watching my 8GB RAM disappear into a vector database just to search my own PDFs. After burning through $150 in cloud costs, I had a weird thought: what if I encoded my documents into video frames?

The idea sounds absurd - why would you store text in video? But modern video codecs have spent decades optimizing for compression. So I tried converting text into QR codes, then encoding those as video frames, letting H.264/H.265 handle the compression magic.

The results surprised me. 10,000 PDFs compressed down to a 1.4GB video file. Search latency came in around 900ms compared to Pinecone’s 820ms, so about 10% slower. But RAM usage dropped from 8GB+ to just 200MB, and it works completely offline with no API keys or monthly bills.

The technical approach is simple: each document chunk gets encoded into QR codes which become video frames. Video compression handles redundancy between similar documents remarkably well. Search works by decoding relevant frame ranges based on a lightweight index.

You get a vector database that’s just a video file you can copy anywhere.

https://github.com/Olow304/memvid
by hn_throwaway_99 | 2025-05-29 04:08:44+00:00
> at some point computer intelligence would overtake biological and here we are, pretty much.

I really hope "pretty much" is doing a lot of work there, because we are still far from the point of computer intelligence overtaking biological. After all, the whole point of TFA is that the AI generated article was full of outright bullshit - it kinda sorta looked plausible, but it wasn't real.

That's the problem with AI - while it definitely is really amazing at some things, in many areas it just seems to have the "mirage" of intelligence.
by pradn | 2025-05-29 04:09:26+00:00
The small writes problem that Iceberg has is totally silly. They spend so much effort requiring a tree of metadata files, but you still need an ACID DB to manage the pointer to the latest tree. At that point, why not just move all that metadata to the DB itself? It’s not sooo massive in scale.

The current Iceberg architecture requires table reads to do so many small reads, of the files in the metadata tree.

The brand new DuckLake post makes all this clear.

https://duckdb.org/2025/05/27/ducklake.html

Still Iceberg will probably do just fine because every data warehousing vendor is adding support for it. Worse is better.
by glenngillen | 2025-05-29 04:10:04+00:00
I once quit a job and gave them 6 weeks notice because I wanted to try and make the handover as painless as possible. My manager said “stop working today. Come in, check your emails, but don’t do any work. Forward and delegate everything that lands in front of you. Let people who reach out to you know you’re leaving and introduce them to whoever is taking over that particular responsibility. We’re going to have to work out how to do things without you. We may as well start today, safe in the knowledge we’ve still got you in the building for the next 6 weeks if we can’t work anything out”.

You can debate the cost effectiveness of the approach, but there was no doubt that by the end of those 6 weeks they were going to be just fine without me.
by tgma | 2025-05-29 04:10:16+00:00
In my experience, Go is one of the best LLM targets due to simplicity of the language (no complex reasoning in the type system or borrow checker), a high quality, unified, and language-integrated dependency ecosystem[1] for which source is available, and vast training data.

[1]: Specifically, Go community was trained for the longest time not to make backward-incompatible API updates so that helps quite a bit in consistency of dependencies across time.
by BrenBarn | 2025-05-29 04:10:45+00:00
> One way to break this illusion is to remember how new things are introduced. Bitcoin didn’t seem more than an intellectual exercise when it was introduced. Facebook seemed like a way to stalk college students. HN seemed like an alternative to Reddit. An iPad seemed like a dumbed-down laptop. Smartphones seemed like a desktop computer in your pocket.

> The point is, once you wait a decade or so and look back, you find that we did in fact get a lot of newness. It just takes awhile to see what makes them distinct from mere optimizations of previous work. AI is no different, and we’re certainly not approaching some singularity moment. Not anytime soon anyway.

If you think that bitcoin and facebook are examples of "real newness" that we only perceive in retrospect, I think we're not seeing eye to eye. Those to me are exactly the kinds of things that represent a colossal waste of human time, effort, and money.
by Caelus9 | 2025-05-29 04:12:51+00:00
Honestly I feel like a lot of the “battery life is worse now” talk kind of misses something important. The way we use our devices has completely changed. Back then we used them in short focused bursts. Now we want our phones and laptops to stay connected all the time, keep background apps running, drive high refresh screens, and even run AI features locally. That’s a big shift in expectations. So maybe battery life didn’t actually get worse. Maybe we’re just expecting way more from our devices than we used to, and we don’t always think about what that really costs.
by kelnos | 2025-05-29 04:13:04+00:00
The presidential election depends on where the borders of each state are, and the population of each state as determined by the decennial census. Gerrymandering does not affect either of those things.

(I otherwise 100% agree that Republicans engage in various forms of voter suppression and disenfranchisement. Not sure what fantasy world GP lives in where they don't.)
by necovek | 2025-05-29 04:14:07+00:00
My point is that what we call conscious and subconscious is limited by our ability to express it in language: since we can't verbalize what's going on quickly enough, we separate those out. Could we learn to verbalize two things at the same time (we all do that as well with say different words and different body language, even consciously, but can we take it a step further? eg. imagine saying nice things to someone and raising the middle finger for someone else behind your back :))

As the whole article is really about the full brain, and it seems you agree our "unconscious mind" producing actions in parallel, I think the focus is wrongly put on brain size, when we lack the expressiveness for what the brain can already do.

Edit: And don't get me wrong, I personally suck at multi-tasking :)
by k310 | 2025-05-29 04:14:12+00:00
What if everything you learned in K-12 were available in an instant? --- except learning how to learn and how to reason through problems --- (not just look up solutions)

Well, we already have the former, even without AI, but I wonder about the latter. I left out "how to express yourself" because we have already reached the "writing event horizon" where it takes considerable "intelligence" to distinguish true self-expression from algorithmic.

I'm not being snide. I grew up through the 4 function calculator, the HP-35, minicomputers, microcomputers, and so on and so forth.

One has to ask "What is education?" I am especially troubled by the post-truth world of unmoderated (anti)social media and by this Matrix world where you can't tell what's genuine and what's algorithmic, with the percentage of material trending quickly towards the latter.

It would seem that the chief skill needed now is the ability to discern fact from fiction, every hour of the day. Case in point. I just looked up products at Amazon and every damn one on the page had a 4.5 star rating. Right.

So, every schoolkid with access to the internet/AI can indeed crank out answers (and tons more) that it used to take a college degree to learn. On the basis of standard exams, they're all straight-A, because they have this "skill" at exam time, and every minute of the day.

So, what's to learn? What the hell do grades mean any more?
by nodesocket | 2025-05-29 04:14:43+00:00
I went back and forth deciding if I should use a service like auth0 / Clerk / Stack Auth for my Flask based app. Ultimately I decided on doing it myself. Storing users and organization in the SQLite db and using a Flask package like Flask-Login. I didn’t want to have to migrate to an ORM (SQLAlchemy) and needed a few basic constraints:

- Users can optionally be assigned to an organization but not required.

- Email address is globally unique.

- A user can only be associated to a single organization.

- 2FA support.

- Single-Sign-On support with Google, Facebook, and GitHub.

I have a feeling not using a framework or auth service is going to bite me in the butt down the line, but it just felt easier to roll my own implementation to start.
by hn_throwaway_99 | 2025-05-29 04:15:14+00:00
Thanks so much for this comment. It's something I've generally felt, but it really didn't crystallize in my head until I read your comment.

As a kid I just remember being enthralled by what the future would bring, and you'd see tons of writing prognosticating about things like "cities of the future" and "houses of the future". I think the fundamental change is that all of those were filled with a sort of techno-optimism. Now, though, I think there is a widespread feeling that tech, as a whole, is no longer in service to the improvement of human society. It just feels like it went off the rails in the past 15-20 years or so, where for a lot of us tech feels like it's made our lives worse.

I no longer look forward to the newest tech or gadget. If anything, I look forward to going for a walk in the woods and leaving my phone at home.
by mmooss | 2025-05-29 04:15:23+00:00
> But the phrase after the semicolon is at the same level as the initial phrase

If you mean my sentence in the GP, here's how I think it parses:

A1 ; A2 . B

The two clauses in the first sentence, connected by the semicolon, are ~equal - but they are subparts of concept A. Concept B is separate and in a separate sentence. If I used no semicolon, I'd have three sentences and there would be no subparts, only

A . B . C
by seltzered_ | 2025-05-29 04:15:27+00:00
"In August 2014, President Obama and Congress created the United States Digital Service (USDS).

In January 2025, President Trump signed an executive order to rename and reorganize it as the United States DOGE Service.

[...]

On March 17 2025, I joined DOGE as a software engineer working for the Department of Veterans Affairs (VA)."

Some important details that should also be mentioned is that:

- "On February 14, 2025, along with other layoffs across the Federal government, several dozen employees who were part of the United States Digital Service prior to the January 20 inauguration of Donald Trump, were dismissed with an email saying, "Due to the restructuring and changes to USDS's mission, USDS no longer has a need for your services."[30][31][32] Of the remaining 65 employees in the department, 21 resigned with a joint resignation letter on February 25 that stated, "We will not use our skills as technologists to compromise core government systems, jeopardize Americans' sensitive data, or dismantle critical public services" (from https://en.wikipedia.org/wiki/United_States_Digital_Service )

- On March 1st, 2025 - " The former employees of 18F said in a joint statement that they had been abruptly locked out of their computers and email at midnight EST, with "no chance to assist in an orderly transition in our work."[28][29] The 18F website was taken offline the same day." ( from https://en.wikipedia.org/wiki/18F#Mass_firings_and_eliminati... )
by truemotive | 2025-05-29 04:15:43+00:00
Cloudflare R3 might suit the scenario better for you in terms of the heavy assets, it's like AWS S3 except for the cool part where you aren't charged for data egress (last I checked! haha)

I'm about to buy Terraria after all these years, just so I can get the assets and check this out. You're cool :)
by zrobotics | 2025-05-29 04:15:54+00:00
I actually specifically made a trip to Milwaukee 1/3 for the Harley museum and 2/3 for ASS (that acronym was definitely deliberate). I've enjoyed my visits to ax-man surplus [0] in Minneapolis enough that when I heard about a similar store it was enough to justify a 500mi weekend bike trip. The stores are very similar, but anyone lucky to live near their store knows that the inventory changes enough that each visit is a unique experience and you won't see the same things twice.

saveitforparts on YouTube did an excellent tour/example video of ax-man [1] which gives a good idea of the type of store ASS is and the vibe. Even if you aren't local to the upper Midwest, I'd highly recommend donating to keep this store alive. It very much fits with the old-school hacker ethos, and keeping spaces like this alive helps preserve that culture and exposes younger people to the idea of DIY/punk/hacker and the ability to repurpose old/interesting stuff.

Side note, I'm surprised that these types of scientific surplus stores have lasted longer in the upper Midwest than places like silicon valley or near bell labs in NJ. They aren't open for retail anymore and didn't have the same sense of humor, but Surplus Sales of Nebraska [2] is also an excellent example from this area of the country. I hadn't ordered since last year, they had held on to a delightful web 1.0 site up until sometime in the last year and are a legit supplier for really oddball stuff.

[0] https://www.ax-man.com/pages/the-nature-of-surplus

[1] https://www.youtube.com/watch?v=U-uJPIhAUu4

[2] https://www.surplussales.com/
by BrenBarn | 2025-05-29 04:15:55+00:00
I think that's only part of the story. Another part is what are you caring about or doing instead. People don't have to care about those things, but what are they doing with their time instead? "Funny fails" videos and getting french fries delivered? Part of the point is that you have to do something with your time, it's just that now people spend their time doing stuff they don't care about, which is sad. You don't have to care about those particular things, but if you're not caring about anything I think it indicates something is out joint.
by lwo32k | 2025-05-29 04:17:27+00:00
Normalcy bias implies Mom can do something about ALL threats one drops in her lap. Sooner or later one realizes Mom can't, and its not related to how loudly one cries or screams about the problem or how quickly Mom runs into the room.

People are acting as if awareness and consciousness is some magic to solving complexity and unpredictability.

Nature has already shown us how to compartmentalize info flows so that we don't get overloaded. For example you are not conscious about what every organ in your body is doing at all time. You are not aware about how many cells are glitching or failing. You are not aware about whats happening at the molecular or atomic level. This is not an accident cause the system doesn't have capacities to do anything about it even if it was magically aware about every problem.

Today our info architecture of society is that pathetically designed, that it is doing nothing useful other than overloading everyone.

People have survived all kinds of chaos throughout history. Just look at the life stories of ordinary people who survived the World Wars or Russian or French revolutions. Guess what helped them from having their brains totally fried by the chaos? Ignorance did. Don't pay attention to things way above your pay grade or career trajectory. Nothing good will happen in the current info landscape.
by rayiner | 2025-05-29 04:17:38+00:00
Legally, there is a major distinction between whether an entity charged with making a finding as indeed done so, and whether that finding correctly reflects the actual facts.

This is not a novel concept. When courts review laws made by Congress, they don’t get to second guess Congress’s economic reasoning. Congress could ban interstate transportation of beer on Tuesdays. As long as the law was within Congress’s commerce power, courts don’t get to second guess the rationale for the law.
by cobyte8 | 2025-05-29 04:18:25+00:00
Hmm, maybe I'm missing something, but you can use the squash workflow so that you don't have to remember to start a new commit: you're always at an empty commit when you context switch to repo.

* If you `git clone ...` and then `jj git init --colocate`, you're in a new commit (whose parent is the HEAD of the git repo).

* If you write some code and then `jj commit`, you're now in a new commit.

* If you want to work on top of commit zyx instead, you can run `jj new zyx` (not `jj edit zyx && jj new`).

* If you're done for the day, you can `jj commit -m "temp: it kinda works, I'll take another stab tomorrow"` and you're on a new commit again.

In many ways, this reflects the normal git workflow (if you want something checkpointed, you've "commit"ed it). This does mean that while I'm in the middle of working on something, I have some commits that aren't split by logic but by chronology. So instead of one commit ("make foo"), I'll have three commits ("try to make foo", "eh, didn't work, try again", "fix bar so that foo actually works now") that reflect my process rather than logic. Once I'm convinced that I finally did it right, jj makes it easy to squash/manipulate the commits into the logical units I want to present in my PR (or record in my history for personal projects).
by jandrewrogers | 2025-05-29 04:20:40+00:00
Most analytics workloads are bandwidth-bound if you are optimizing them at all. The major issue with disaggregated storage is that the storage bandwidth is terrible in the cloud. I can buy a server from Dell with 10x the usable storage bandwidth of the fastest environments in AWS and that will be reflected in workload performance. The lack of usable bandwidth even on huge instance types means most of that compute and memory is not doing much — you are forced to buy compute you don’t need to access mediocre bandwidth of which there is never enough. The economics are poor as a result.

This is an architectural decision of the cloud providers to some extent. Linux can drive well over 1 Tbps of direct-attached storage bandwidth on a modern server but that bandwidth is largely beyond the limits of cheap off-the-shelf networking that disaggregated storage is often running over.
by rayiner | 2025-05-29 04:22:11+00:00
> The acts give him the power to enact tariffs under certain conditions

Right, but the “condition” in the law is that the President first makes a “finding” that other countries have engaged in unfair treatment. The president only needs to convince the court the finding has been made. But whether a trade deficit results from trade barriers or something else is a decision that Congress has delegated to the President to make.
by throwawaymaths | 2025-05-29 04:23:10+00:00
you can make an evidenced argument that elected officials of the Democratic party have been systematically destroying public education for decades (for example cancelling algebra in California, repeat child abusers not let go in LAUSD, see "mark berndt" for an example of a multidecade offender that bounced between schools)

really, Republicans arent helping, but they arent necessary to destroy American schools thats happening just fine in ~one-party democrat states like California and hawaii.
Show HN: Typed-FFmpeg 3.0–Typed Interface to FFmpeg and Visual Filter Editor
by lucemia51 | 2025-05-29 04:23:11+00:00
Hi HN,

I built typed-ffmpeg, a Python package that lets you build FFmpeg filter graphs with full type safety, autocomplete, and validation. It’s inspired by ffmpeg-python, but addresses some long-standing issues like lack of IDE support and fragile CLI strings.

What’s New in v3.0: • Source filter support (e.g. color, testsrc, etc.) • Input stream selection (e.g. [0:a], [1:v]) • A new interactive playground where you can: • Build filter graphs visually • Generate both FFmpeg CLI and typed Python code • Paste existing FFmpeg commands and reverse-parse them into graphs

Playground link: https://livingbio.github.io/typed-ffmpeg-playground/ (It’s open source and runs fully in-browser.)

The internal core also supports converting CLI → graph → typed Python code. This is useful for building educational tools, FFmpeg IDEs, or visual editors.

I’d love feedback, bug reports, or ideas for next steps. If you’ve ever struggled with FFmpeg’s CLI or tried to teach it, this might help.

Thanks! — David (maintainer)
by magicalhippo | 2025-05-29 04:23:13+00:00
Quality isn't a single thing though.

Being reasonably sure you get the same thing when you order again is a kind of quality, which McDonalds has. So in that aspect, McDonalds provides a quality service. Not something I can say about my local kebab joint, where I have to consider who's working that day.

And the kind of quality that sets Maybach cars apart, is not something most people want to spend money on. Being able to produce a good-enough tool for a low price is one kind of quality, which many customers considers attractive.

That's where we're at at work. Our niche is serving a low-margin market, so customers are very price sensitive. They can tolerate some non-critical bugs here and there, if it means they don't have to pay as much.

If we were to follow Saarinen's path and only ship fully completed features or fix all bugs within 7 days, we'd most definitely would need twice as many technical people, which would significantly impact the price we'd have to charge.

That said, I think his other points, especially seeking to understand our customers and having small teams, is something that's worked very well for us as well. This has been aided by hiring folks from our customers in support roles, helping us to internalize domain experience.
by throwaway5752 | 2025-05-29 04:23:15+00:00
I generally agree. If I had been ask to review this, I would have pushed very, very hard on whether "cosmological natural selection assumes that universes reproduce via black holes and big bangs" and most of what follows was necessary. Even if it's true I wouldn't have mentioned that Tyler Cowen was solicited for a review. And not having any math, at all, makes me think the author is not committed enough to their own idea to put in the hard work.

It was a very interesting piece, and something to be proud of. So far nobody has figured out the TOE so they are in good company. On the other hand I see no reason for them to believe they are in a position to be as dismissive as they are of existing theory.
by palmotea | 2025-05-29 04:24:28+00:00
> Old companies die and new ones take their place. This is not a problem for society. In most cases creative destruction is a net positive.

That's what they say, but I don't think it's true (at the high end, at least). For instance: if Boeing dies, the market will not replace it. It'll be an Airbus monopoly for large jets, and maybe the the communists will eventually build a competitor (Comac). IIRC, it's too expesnive for Embraer to make the jump into that market.
by nitwit005 | 2025-05-29 04:24:52+00:00
If you make the brain larger, some things will get worse, rather than better. The cost of communication will be higher, it will get harder to dissipate heat, and so on.

It's quite possible evolution already pushed our brain size to the limit of what actually produces a benefit, at least with the current design of our brains.

The more obvious improvement is just to use our brains more. It costs energy to think, and for most of human existence food was limited, so evolution naturally created a brain that tries to limit energy use, rather than running at maximum as much as possible.
by api | 2025-05-29 04:25:01+00:00
It may have been back then, though I'd argue that you could have done it back then with efficient code and a very well-tuned DB.

Today big boxes are big. Really big. Stuff like 128 cores, 1TB RAM, and dozens of terabytes of incredibly fast RAIDed flash storage is available out there.

They're also more reliable than they used to be. Hardware still fails of course, but it doesn't fail as often as OG spinning disk did.