MurphyA/hacker-news huggingface dataset:
by jsd1982 | 2025-05-31 15:04:04+00:00
The obvious solution to me is to implement streaming/buffered processing of the content while downloading it instead of downloading the entire content into memory to be processed in a single contiguous byte[].

Buffered IO would have the CPU processing acting as a natural backpressure mechanism to prevent downloading too much content and also prevent unbounded memory allocation. Each CPU worker only needs to allocate a single small buffer for what it processes and it can refill that same buffer with the next IO request. Your memory usage becomes entirely predictable and will only scale with how many concurrent threads you can actually execute at once.

Also, no matter how you artificially rate limit the virtual thread scheduling (e.g. via semaphore), if you still insist on downloading the entire content into memory before starting processing then obviously you cannot process any single piece of content larger than what can fit into available memory at any given time.
by pathsjs | 2025-05-31 15:04:34+00:00
In fact, this is an opportunity to anticipate what is going to be written. I studied maths, so I am not sure how it would work for other fields of study. But one thing that worked very well for me was to write everything, spoken or written, before it was said, or to try to stay 10 seconds ahead of the lecture. This would keep my attention and it would require an active comprehension, like continuously doing exercises. I was then able to recall what was said much better. And I could take advantage of the pauses to get a little ahead so I would not get lost in the next difficult passage. Of course, this only works in fields where there is a strong logical connection between the various parts of the lecture, so that you can indeed try to figure out yourself, it would not work where there is a lot of memorization
by spankibalt | 2025-05-31 15:04:35+00:00
> I know many artists that are not thirsty for competition.

Good for you.

> Actually I don't know any passionate artist who would create with competition in mind.

I on the other hand know many; the demosceners I grew up with, just to give one example. Competition there is part of these subcultures' very own aesthetics, i. e. (unwritten) rules.

> If that is your definition of "artists" than you are talking about content creators.

Albeit I have an aversion for the slopword "content" and avoid it to describe any artist's output, artists are, as creators (of "content" of an artistic nature) per se, i. e. per definition, "content creators". You also erected a strawman, for I only adressed competency as an attribute of definition... and not competition. And albeit the latter might help make anyone more competent as an artist, or empower an artist in becoming one in the first place, the concept doesn't define the artist as one on its own, in itself.

And with that I think we have exhausted our exchange at this point as a) I have answered your question about AI features in art packages, and b) you already have trouble understanding even the most basic of concepts in art creation, let alone art theory.
by karmakaze | 2025-05-31 15:04:57+00:00
It depends on how you look at it. If the output 'it' is a drawing, then the agent is the thing doing the drawing on the user's behalf. In more detail the output thing are commands, so then the agent would be what's generating those commands from the user's input. E.g. a web browser is a user agent that makes requests and renders resources that the user specifies.
by OtherShrezzing | 2025-05-31 15:05:12+00:00
>I don’t understand why people are ok with Google scraping their site (when it is called indexing), fine with users scraping their site (when it is called RSS reading), but suddenly not ok with AI startups scraping their site.

Google and the publisher share the value derived from that scraping event (somewhat equitably) when a user clicks a link in Google's index. OpenAI, Anthropic, and co extract all of the value, and incur costs to the publisher, from their scraping events.
by atiedebee | 2025-05-31 15:05:27+00:00
It's funny how they say they are at 10% of the code of LLVM in their marketing, when it is actually closer to 0.1%. Throwing LLVM version 20.1 through wc (just the subdirectory llvm/lib) results in 2.3 million lines for all .c and .cpp files. QBE was around 14k last time I checked.
by sokka_h2otribe | 2025-05-31 15:06:03+00:00
This was largely driven by the efficiency and fuel density.. so like, the mines specifically were coal mines which flood and need pumps, but obviously have unlimited coal. The engines/tractors were competing against oxen and horse, and it's hard to get a lot of power out of them. The trains had an easy time to carry lots of coal due to their nature.

I'm not actually sure where tractors fit in. I haven't heard of them in the equation early on. I think at some point they were probably viable, but I never heard of a coal powered tractor (maybe there were some). I suppose tractors could leave piles of coal and stuff if they needed to by the fields.
by bhl | 2025-05-31 15:06:15+00:00
Ecosystem for local-first and mobile is pretty immature, at least for Swift.

In comparison to the web where there's so many libraries e.g. Zero, LiveStore, LiveBlocks, I've yet to find a good GRDB (sqlite abstraction) integration / client.

Offline-first is definitely very strong, but now how do I get data into a remote database with conflict resolution support?
by pornel | 2025-05-31 15:06:17+00:00
C++ syntax and semantics are optimized for C++ idioms, and Rust's aren't.

Rust is more related to ML-family languages than C-family and OOP, so it needs to "emulate" some C++ idioms.

This goes both ways, e.g. equivalent of Rust's pattern matching on enums with data translates to verbose and clunky C++.
by pron | 2025-05-31 15:06:26+00:00
The CPU is only contended once it's at 100%. I am not aware of any interactive server (as opposed to batch processing) that behaves well at 100% CPU (beyond short bursts), so I think that the hope you can actually balance the CPU using the OS scheduler (which isn't all that good) to produce good outcomes at 100% CPU is more myth than reality. So yes, I'm talking about resources accessed via IO, because there isn't all that much you can do about a CPU that's at 100% for long durations other than getting more CPU (horizontally or vertically).

However, when you have some background batch computational tasks of some limited concurrency (otherwise you'd be in trouble) then you should have some low-priority platform threads servicing that. Virtual threads work due to Little's law by allowing a high number of threads. For such low-priority background tasks you probably want a very low number of threads (often one), so the ability to have tens of thousands, or millions, of virtual threads won't help you for those background tasks anyway.
by cratermoon | 2025-05-31 15:06:56+00:00
> They also respect robots.txt

All the reports I've heard from organizations dealing with AI crawler bots say they are not honest about their user agent and do not respect robots.txt

"It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more." https://xeiaso.net/notes/2025/amazon-crawler/
by mushufasa | 2025-05-31 15:07:23+00:00
I actually started my startup with this approach -- good old Django.

Lasted for a year until we had users wanting to do complex interactions on the frontend.

Simple CRUD apps are fine, but users expect a lot of interactivity these days where things on the screen interact with each other.

The nice thing about Django, though, is that you can easily switch to api-based backend with DRF or Django-Ninja. We used DRF but if I were to do it again, I'd choose ninja.
by lolinder | 2025-05-31 15:07:54+00:00
> The example scenario isn't "person asks the waiter for a hot dog; waiter just keeps walking past them, blanking them as if they don't even exist".

First: that scenario is actually still just fine when the establishment is a soup kitchen. When the people providing the service are providing it for free funded entirely out of their own pocket then no user of the free service is entitled to anything, ever.

Second: that isn't an accurate description of what happened, as you should know because you read the thread in full. What actually happened is that a maintainer responded to the issue back in January with a soft timeline. OP began harassing that maintainer less than halfway through the provided timeline.

When they got ignored for just over a week (which is, again, entirely reasonable especially given that a timeline was already offered) OP proceeded to write not one but three "FUCKUJimmy" acrostics further harassing the maintainer who had provided the initial timeline.

Then, on May 9, which is well within the initial estimate, the maintainer provided an update saying that the timeline had to be pushed back. OP's reaction to that was to try to bring the project hate on HN.

In what universe is any of this behavior by OP acceptable?
by latchkey | 2025-05-31 15:08:20+00:00
This really brings back memories. The first computer language I learned as a child was Logo. My grandfather gifted me a lesson from a local computer store where someone came out to his house and sat with me in front of his Apple II.

I was too young to understand the concepts around the math of steps or degrees. While the thought of programming on a computer was amazing (and later became an engineer), I couldn't grasp Logo, got frustrated, and lost interest.

If I could have had something like this, I'm sure it would have made more sense to me earlier on. It makes me think about how this will affect the learning rate in a positive way.
by vasco | 2025-05-31 15:08:41+00:00
Why 175? Why not 5 billion employees? Why not 20000 companies in parallel? Why not simulate 5 earth's worth of history and setup a full universe of worlds full of startups?

This sounds like those guys in social media that one up each other with their bed times and end up saying they wake up every day at 2am to meditate and work out
by ndriscoll | 2025-05-31 15:09:14+00:00
It should be, though it would be nice if browsers let you do enctype=json (and maybe enctype=xml). Json is popular enough at this point that you're more likely to have a high performance, easy to use codec library on the backend vs. FormData.

Form validation probably should have also been done via child tags, not attributes, so that error messages could be independently associated to each validation rule.
by mnky9800n | 2025-05-31 15:09:31+00:00
I thought to myself, imagine something you’ve never imagined before. My first thought was what if there is a universe inside of every vegetable that is vegetable themed with anthropomorphic vegetable characters and all the atoms and molecules are some how veggified and everything is a vegetable. And then I wondered if an AI could ever come up with that with infinite time and resources without a prompt and then I thought about monkeys and typewriters.
by michaelmarkell | 2025-05-31 15:09:39+00:00
From the Readme:

Given a model with pre-trained transcoders, it finds the circuit / attribution graph; i.e., it computes the direct effect that each non-zero transcoder feature, transcoder error node, and input token has on each other non-zero transcoder feature and output logit. Given an attribution graph, it visualizes this graph and allows you to annotate these features. Enables interventions on a model's transcoder features using the insights gained from the attribution graph; i.e. you can set features to arbitrary values, and observe how model output changes.

The blog post: https://www.anthropic.com/research/open-source-circuit-traci...
by karim7793 | 2025-05-31 15:09:45+00:00
I plan to add documentation in several languages shortly. However, the tool itself has a user menu that is in English by default; only the readme is in French. To answer the question, the webscanner reads the headers and displays various information about the files and available endpoints of the scanned website. For reconnaissance, it performs a whois lookup and several other checks. OSINT can find email addresses associated with the target site, while also performing many other checks. As for the hash cracker, it has undergone a major patch, making it fully usable without any bugs or issues.
by pfdietz | 2025-05-31 15:10:01+00:00
The current global CO2 emission rate is about 200x the average emission rate during the Permian/Triassic mass extinction. Just slowing down these emissions buys time, which is useful and valuable, but is not itself a solution. Net CO2 emissions have to be driven effectively to zero or we're just delaying disaster.

Also, "net emissions" that include CO2 being absorbed into the oceans must take into account that oceanic absorption declines as surface waters become more saturated with dissolved CO2 and the pH declines. The rate of carbon flux to the deep ocean (from descending organic detritus) is just a few percent of annual emissions.
by 1dom | 2025-05-31 15:11:08+00:00
I agree, modern browsers, CSS and JS can do so much now.

However, the frontend build systems are often complex to do complex state management. Sure, it's possible to reimplement it all and accidentally end up rewriting your own vanilla JS frontend framework, but beyond a point, that's equally as unfun as modern webdev.

It's not hard to shift the state management though, and when I find myself sat with a nice architecture that's formed without a complex frontend system, and with some really simple, uncluttered backend that could probably serve millions of requests a day from a homeserver and internet connection.... it gives me a profound feeling. It's a satisfying and familiar feeling from webdev I enjoyed around 2 decades ago as a teen. It feels like the old way of doing it.

Now, having designed, built and run systems from on-prem, through VPS & VMs onto serverless and static site stuff, doing it the old way again just feels better overall. Maybe it's nostalgia.

But then I feel the reason why we moved away from doing it the old way was because the complexity of the applications needed more efficient ways to make use of relatively limited bandwidth, storage and compute resources at the time. Now these things are all commoditised, and my home desktop and internet connection now has probably the same capabilities as a small datacentre back then... the dreamer in me wants to believe there's no reason why we can't all go back to the old ways of doing it now.
by sillyfluke | 2025-05-31 15:11:11+00:00
Yeah, I vaguely remember dang saying that they might consider allowing users to unflag posts, but I think he did confirm that currently flagging cannot be undone by non-moderator users. Currently the site errs on the side of flagging for that reason. I think there was a tweet from pg saying that they were talking about the possibility of giving unflagging rights to karma users with dang, but I'm going completely off memory here.
by p0w3n3d | 2025-05-31 15:12:21+00:00
If wind has kinetic energy - E, the amount that can be harvested is < E, and that was kinda obvious from the start. Turbines stop the wind, hence one country can stop wind blowing towards another country if they put many fields on the "track". And apparently it hasn't been taken into account when making those optimistic statements regarding zero emission energy. Also this IS ecology altering, and might have an effect in global weather, but not sure how much (wind mainly blows at high altitudes).
by justonceokay | 2025-05-31 15:12:22+00:00
A pet issue I have that is in line with the “illusions” in the article is what I might call the “bound by statistics” fallacy.

The shape of it is that there is a statistic about population and then that statistic is used to describe a member of that population. For example, a news story that starts with “70% of restaurants fail in their first year, so it’s surprising that new restaurant Pete’s Pizza is opening their third location!”

But it’s only surprising if you know absolutely nothing about Pete and his business. Pete’s a smart guy. He’s running without debt and has community and government ties. His aunt ran a pizza business and gave him her recipes.

In a Bayesian way of thinking, the newscasters statement only makes sense if the only prior they have is the average success rate of restaurants. But that is an admittance that they know nothing about the actual specifics of the current situation, or the person they are talking about. Additionally there is zero causal relationship between group statistics and individual outcomes, the causal relationship goes the other way. Pete’s success will slightly change that 70% metric, but the 70% metric never bound Pete to be “likely to fail”.

Other places I see the “bound by statistics” problem is in healthcare, criminal proceedings, racist rhetoric, and identity politics.
by hulitu | 2025-05-31 15:12:34+00:00
> I thought the combination of hardware+OS by same company would lead to a better experience but nope.

It is the same with their software. It seems that every team is working independently. They are basically not able to release a "distribution" due to this. I wonder how many versions they need to have a unified look and feel like they in Windows 3x and later in NT4 to Win 7.
by milocoombs | 2025-05-31 15:12:58+00:00
This is a recent piece of independent research I did exploring a new formulation of neural networks as continuous integral operators, inspired by variational principles in physics. My ideas went viral on LinkedIn.

Instead of stacking discrete layers, this framework uses a weight function W(s,t) that maps input functions to output functions via integration. Learning is framed as minimizing an action functional composed of kinetic and potential energy terms, much like in classical mechanics.

It leads to an integro-PDE for the optimal weight function, which invites interesting analytical and numerical techniques from physics and applied math. The framework is especially promising for scientific computing and signal processing, though it's still early-stage.

Would love feedback or thoughts from others exploring functional or physics-based ML models. Link is to the full write-up.
by jfengel | 2025-05-31 15:13:22+00:00
I appreciate that, and science should be saved.

But the cuts go far beyond that, to things that are less showy but with more direct effects on the lives of people. Tens of thousands of jobs have been lost, with microscopic effects on the bottom line, but with myriad small effects on our parks, our roads, even national security.

Space is the "charismatic mega fauna" of the budget, but the whole "ecosystem" is ravaged. It must be saved, but take it as an indicator of just how much damage has been done. Spare a thought and a word for the NIH, the Fish and Wildlife Service, and the Social Security administrators whose jobs are not glamorous but nonetheless critical to people who depend on them.
by pydry | 2025-05-31 15:13:38+00:00
>or base load power plant, like nuclear, will continue to have marketable "goods".

Nuclear hasnt been marketable for a long time without massive indirect and direct subsidies.

The reason it gets built in spite of its demand for lavish subsidies is because it shares a supply chain and skills base with the military industrial complex.

Countries that have a nuclear military want to share some of the costs with the civilian sector although they typically arent up front about it, preferring to declare that it's because they're environmentalists.

NPT signatories that dont have nuclear weapons but take a strong interest in building civilian nuclear power typically see some sort of potential existential risk on the horizon. Again, officially it's because theyre green hippies.

In unrelated news, after decades of being by far the most coal addicted country in Europe: https://apnews.com/article/poland-us-nuclear-energy-power-pl...
by bediger4000 | 2025-05-31 15:14:04+00:00
In the early days of blogging, I read some post from a guy who consulted at Microsoft. He wrote about how he had to sign the regular sort of NDA, and a second, horrifyingly strict NDA about how he wouldn't disclose anything he learned about Clippy. This was during the period when Clippy was working at a re-education farm, before getting rehabilitated. I have looked for this post several times, never finding it again.
by vsupalov | 2025-05-31 15:14:52+00:00
I'm also curious about ways to provide value with a technical project.

The challenge when exploring this topic: the incentive to stay under the radar. Those succeeding don't have much to gain from sharing details here. Worst case: it could invite competitors into their space.

Communities that thrive on growth (e.g., open-source) tend to share freely, but API businesses, especially ones which are easy to execute, often guard their edge.

A recent finding I had, while not necessarily an API: services which help you 24/7 stream a lenghty video file. YouTube live streams seem to work well for those lofi-types of channels, and there are services which are built to enable autopilot live streams.
by bubblyworld | 2025-05-31 15:15:04+00:00
Yeah, as an example this is a simple corollary of the following result which gets used a lot: the quotient of a (unital, commutative) ring by a maximal ideal is always a field.

If your ring has only two ideals then the trivial ideal is maximal, and thus your ring is already a field!

The more you know, the more "shortcuts" you start seeing, I guess.
by ath3nd | 2025-05-31 15:15:58+00:00
> A lot of software engineers can get to $5M-$10M range in like 10-30

$5-$10M for 30 years, but only if you save every penny in between? Wow, that's very impressive and totally life-changing! Reminds me of the story how millennials are not able to afford buying a house because of avocado toast!
by dataflow | 2025-05-31 15:16:06+00:00
> That your position hinges on thinking all languages share the same memory model suggests a much deeper failure to understand some of the basic principles of writing correct parallel software and while numerous people have tried to correct you on this, you still seem adament on doubling down on your position so I don't think there is much point in continuing this.

I never suggested "all languages share the same memory model". You're severely mischaracterizing what I've said and putting words in my mouth.

What I said was (a) data races are generals properties of programs that people can and do discuss in language-agnostic contexts, and (b) it makes no sense to say two well-defined, equivalent programs differ in whether they have data races. Reducing these statements down to "all programs share the same memory model" as if they're somehow equivalent makes for an incredibly unfaithful caricature of everything I've said. Yes, I can see there's no point in continuing.
by cipherboy | 2025-05-31 15:16:20+00:00
While I'm sure Vault contracts run more than what I'd care to know, the project is set up under the Linux Foundation and I've been told in the past that we as a project are capable of receiving direct donations.

If you're so inclined, please do!

We'd of course be appreciative of it, but that said, the OpenBao TSC had tabled conversations about just what we'd spend any funds on until after we moved into the OpenSSF... Which just concluded which means it might be time to get things moving again. But just to say, we may not immediately know what we'd spend any donations on. :-)

(Alternatively, hiring and retaining a maintainer or a firm working on it would also be good options. Part of the growth requirements of OpenSSF projects is to have more than a handful of companies on project leadership so increasing diversity is a key goal.)
by chris_wot | 2025-05-31 15:16:23+00:00
If you remove the rule of law, then you let tyrants reign. Best of luck with the complete breakdown of your separation of powers - it’s why you have your freedoms.

You’ll care when you need the government to heed a court decision taken against them and they completely ignore it. At that point though, it’s too late.

Incidentally, many of those who were deported did have the consent of the courts to remain.