MurphyA/hacker-news huggingface dataset:
by koito17 | 2025-11-05 02:08:53+00:00
12 TB fits entirely into the RAM of a 2U server (cf. Dell PowerEdge R840).

However, I think there's an implicit point in TFA; namely, that your personal and side projects are not scaling to a 12 TB database.

With that said, I do manage approximately 14 TB of storage in a RAIDZ2 at my home, for "Linux ISOs". The I/O performance is "good enough" for streaming video and BitTorrent seeding.

However, I am not sure what your latency requirements and access patterns are. If you are mostly reading from the 12 TB database and don't have specific latency requirements on writes, then I don't see why the cloud is a hard requirement? To the contrary, most cloud providers provide remarkably low IOPS in their block storage offerings. Here is an example of Oracle Cloud's block storage for 12 TB:

Max Throughput: 480 MB/s Max IOPS: 25,000

https://docs.oracle.com/en-us/iaas/Content/Block/Concepts/bl...

Those are the kind of numbers I would expect of a budget SATA SSD, not "NVMe-based storage infrastructure". Additionally, the cost for 12 TB in this storage class is ~$500/mo. That's roughly the cost of two 14 TB hard drives in a mirror vdev on ZFS (not that this is a good idea btw).

This leads me to guess most people will prefer a managed database offering rather than deploying their own database on top of a cloud provider's block storage. But 12 TB of data in the gp3 storage class of RDS costs about $1,400/mo. That is already triple the cost of the NAS in my bedroom.

Lastly, backing up 12 TB to Backblaze B2 is about $180/mo. Given that this database is for your dev environment, I am assuming that backup requirements are simple (i.e. 1 off-site backup).

The key point, however, is that most people's side projects are unlikely to scale to a 12 TB dev environment database.

Once you're at that scale, sure, consider the cloud. But even at the largest company I worked at, a 14 TB hard drive was enough storage (and IOPS) for on-prem installs of the product. The product was an NLP-based application that automated due diligence for M&As. The storage costs were mostly full-text search indices on collections of tens of thousands of legal documents, each document could span hundreds to thousands of pages. The backups were as simple as having a second 14 TB hard drive around and periodically checking the data isn't corrupt.
by eru | 2025-11-05 02:10:34+00:00
Well, that's what you get for not running a totalitarian country.

By and large, it's a feature, not a bug, that the government isn't sharing all the information it has between its various parts.

Eg GCHQ has lots of information it has (ostensibly) for keeping the country safe, but that doesn't mean that the prosecution in a criminal case should get access to all the same information.

Of course, that's a bit inefficient and duplicates efforts. But such is a price for restrained government.
by gus_massa | 2025-11-05 02:13:54+00:00
Clicky: https://colab.research.google.com/github/Zen-Sherbert/Proteu... https://github.com/Zen-Sherbert/Proteus-Attention/tree/main

> The idea is simple: chunking. Break a massive context into small pieces, shunt them to system RAM, and use a tiny VRAM buffer for only the most important tokens.

So, ... you are cherry picking some tokens to be added to the context?
by fifilura | 2025-11-05 02:15:39+00:00
How do you use your data lake? For me it is much more than just storing data, it is just as much for crunching numbers in unpredictable ways.

And this is where postgres does not cut it.

You need some more CPU and RAM than what you pay for in your postgres instance. I.e. a distributed engine where you don't have to worry about how big your database instance is today.
by arc_of_justice | 2025-11-05 02:16:07+00:00
Location: Eastern USA

Remote: Yes

Hybrid: Yes

100% On-site: Maybe

Willing to relocate: No

Technologies: Python, PySpark, Pandas, Polars, Scikit-learn, Seaborn, Altair, Setuptools, uv, MySQL, Postgres, Athena, Snowflake, AWS, Azure, IaC(AWS CDK + Terraform), Docker, Unit Testing, Jenkins, Atlassian, InfluxDB, Grafana, HomeAssistant, OpenWRT, LinuxAdministration, R, C, C++, Rust, and Scala.

Email: 66_disco.scenery@icloud.com

Resume: https://docs.google.com/document/d/1PDweThDyDtRe5sU2h7bxVIhp...? usp=sharing&ouid=101518583569305572349&rtpof=true&sd=true
by eru | 2025-11-05 02:17:32+00:00
Btw, long distance busses have a really strange history in Germany.

The literal Nazis made a law that virtually banned long distance busses inside of Germany, and the market was only liberalised in 2013. Deregulation and liberalisation often get a bad rep, but they have done a lot for us.

(To be more precise, the Nazis didn't outright ban long distance busses directly, what they did was give the government railway monopoly a veto over most bus routes and lots of extra restrictions. Which amounted to the same result as a ban. Just like the US doesn't directly ban buying from the world's most popular electric car brand or importing photovoltaic cells: they just slap outrageous 100%+ tariffs on them.)
by artdigital | 2025-11-05 02:19:02+00:00
I’m not paying for the human that made the app. I’m paying for the app, aka for an advertised thing with an advertised feature set for a specific price. If I deem the value I get for the price worth it, I will purchase the thing from you.

I will however not pay you monthly just because “the dev needs to eat too” if there is no service provided that justifies the monthly ongoing cost.
by eru | 2025-11-05 02:20:29+00:00
And that's a big shame!

You can set up your tax system so that you raise a lot of public money from foreigners using your real estate. Be that Chinese tycoons parking their money in London apartments, or digital nomads renting a large house in the countryside.

It's really straightforward: a high enough property tax will do it automatically. Or if you want to get fancy, use a land value tax, which is like a property tax, but you get a discount for the value of the building.
by busterarm | 2025-11-05 02:21:49+00:00
Still missing the point. This is just one server and just in the dev enviornment?

How many pets do you want to be tending to? I have 10^5 servers I'm responsible for...

The quantity and methods the cloud affords me allow me to operate the same infrastructure with 1/10th as much labor.

At the extreme ends of scale this isn't a benefit, but for large companies in the middle this is the only move that makes any sense.

99% of posts I read talking about how easy and cheap it is to be in the datacenter all have a single digit number of racks worth of stuff. Often far less.

We operate physical datacenters as well. We spend multiple millions in the cloud per month. We just moved another full datacenter into the cloud and the difference in cost between the two is less than $50k/year. Running in physical DCs is really inefficient for us for a long of annoying and insurmountable reasons. And we no longer have to deal with procurement and vendor management. My engineers can focus their energy on more valuable things.
by fifilura | 2025-11-05 02:21:59+00:00
It is not hyperbole. I'd say it is in the area of microservices and data crunching. Where 90% of the code is about ingesting and forwarding.

Depends on your viewpoint. For some a car is just 4 wheels that take you somewhere. For a mechanic it is a world full of valves and pipes.

But I hope not every team builds a car from scratch.
by eru | 2025-11-05 02:22:05+00:00
It depends. And different people care about different things.

Eg some people complain about tourists renting AirBnBs. Some people complain about digital nomads competing for rental apartments.

As you suggest, lots of other people are happy for tourists and digital nomads to come in and bring money to the local economy.

And some people are happy to receive the benefits, and still complain about the drawbacks. People really like to complain!
by thedufer | 2025-11-05 02:22:33+00:00
> if you reduce latency from one second to a hundred milliseconds, could you celebrate that you've made it 10x faster

Yes you can, because speed has units of inverse time and latency has units of time. So it could be correct to say that cutting latency to 1/10 of its original value is equivalent to making it 10x the original speed - that's how inverses work.

Savings are not, to my knowledge, measured in units of inverse dollars.
by Marshferm | 2025-11-05 02:23:51+00:00
Details are bigger than sales pitch headline “Dr. Areán said many in the mental health field have lingering concerns about the text model, because the therapist is not face to face with the patient and may miss danger signs. But there were no adverse events in the trial, she said. The trial participants were more likely to drop out of video-based therapy than text-based. But at the same time, the researchers reported, the video sessions appeared to create a slightly stronger bond with therapists, who were viewed by patients as warmer and more compassionate. When some participants switched from text-based to video sessions in a second phase of the trial, the quality of the bond improved significantly. Dr. Areán said the gap in therapeutic bonding merited further research. “How do you build alliance when you can’t see the person?” she said. “How do you deal with silences when you can’t see them? Are they crying? Are they happy?”
by oidar | 2025-11-05 02:25:07+00:00
His lone comment:

>Yes it is true: the second word of a TIFF file, 42, was indeed taken from the answer to the Ultimate Question of Life, the Universe, and Everything, from Hitchhikers_guide_to_the_galaxy. StephenECarlsen 23:38, 12 October 2007 (UTC)
by kouteiheika | 2025-11-05 02:25:46+00:00
It ticks a checkbox. That's it. Some organizations and/or governments might have rules that emphasize using international standards, and this might help with it.

I just hope it's going to be a "throw it over the fence and standardize" type of a deal, where the actual standardization process will still be outside of ISO (the ISO process is not very good - not my words, just ask the members of the C++ committee) and the text of the standard will be freely licensed and available to everyone (ISO paywalls its standards).
by embedding-shape | 2025-11-05 02:26:04+00:00
> suffered an unrecoverable mechanical failure

But was un-discoverable? Or un-preventable? Seems plane inspectors and safety-related roles were affected and have been furloughed:

> But for the people involved in inspecting our planes to ensure they follow Federal Aviation Administration safety standards, the situation is more complicated. While principal aviation inspectors were told to keep working, assistant-level inspectors and other support staff were sent home and then had to be recalled.

https://archive.ph/rEpTx
by bawolff | 2025-11-05 02:26:29+00:00
My point is more - if you make inheritence have a large tax but just giving things to family members a small tax, people will just gift their assets before death.

Any tax plan that requires people to be stupid in order to be taxed is a bad one because then the only people who get taxed are poor people who can't afford to hire an accountant to figure these things out for them.

The obvious way to close that loophole would be to tax capital gains at the same way as inheritence, but the side effects of that seem disasterous.

> I don't see the downside unless you think it's fine for children to simply millionaires for being related to one.

I think the downside is that the actual affect of the plan would be that really rich people would be unaffected, and if anyone was actually taxed it would be low-mid income people. Thus further cementing inequality.
by EdwardDiego | 2025-11-05 02:27:10+00:00
I wrote a lot of comments - for humans, but then I also specifically addressed some to the bot - "Cursor, remove all emojis in log messages, and do not use print for debugging, use a logger, where you are using a logger you are repeatedly importing the logging library in various conditional clauses, you should always import at the top level of the file" etc. etc. etc. - because you know that they're going to feed my review back to the bot.

The fact that someone submitted this PR in that state though...
by eru | 2025-11-05 02:28:33+00:00
Despite the downvotes you are right that democracies mostly deliver what voters want. And if voters want silly things, they get silly things.

It's easy to fret at how dysfunctional and insane politics are. But after you talk to some actual voters (and look at opinion polls), instead you marvel at how comparatively sane policies manage to be---despite voters.
by andrehacker | 2025-11-05 02:28:50+00:00
Am I missing something ?

The article is great but the web site is supposedly related to a book "inventing the future".. which is nowhere to be found. Other than a big, slowly loading graphic, 3 posts and indexes for the book... the site doesn't provide a clue about where to acquire the actual (PDF only?) book.

I assume you have to sign up to find out more ?

On the web I can only find articles about the book.

So.. what is the deal in making the actual book hard to find ?

Edit: I think I cracked the code: Click Home, Open "Close Your Rings" article, scroll all the way down, find link: https://books.by/john-buck?ref=inventingthefuture.ghost.io
by digianarchist | 2025-11-05 02:30:46+00:00
No freedom of press, no freedom of speech, no freedom of assembly, government owns/operates roughly ⅓ of the economy that features many state monopolies, the PAP maintains a gerrymandered control over the electorate, criticizing the government lands you in court for defamation and conveniently bankrupts can't run for parliament.

Singapore is many things but not none of what you've written.
by Daneel_ | 2025-11-05 02:31:03+00:00
From memory there was no dedicated event separator, it just went straight from the last character of the event to the first character of the timestamp of the next event. I think there was an XML payload in the event somewhere too?

Fortunately I didn't have to edit the log in-place as we were ingesting it into Splunk, so I just wrote some parsing configuration and Splunk was able to munch on it without issue.
by vibrio | 2025-11-05 02:31:16+00:00
“He had the temerity to reject a drug that had lousy data…”

Was that data really “lousy”? (Referencing the REPL data?) Was it a trial design issue? (which he has very strong and unconventional opinions on) Is it the role of his position to overrule his specialist review teams ? (in the absence of any clear safety risks or malfeasance)
by filleduchaos | 2025-11-05 02:31:24+00:00
Taking off with one engine inoperative (on a multi-engine aircraft, obviously - you aren't going to get anywhere with your only engine gone) is completely normal/within design parameters, albeit undesirable.

In fact, it being normal almost certainly contributed to the scale of this accident, since a single engine failure during the takeoff roll isn't considered enough of an emergency to reject the takeoff at high speed (past a certain speed, you only abort if the aircraft is literally unflyable - for everything else, you take the aircraft & emergency into the air and figure it out there). The crew wouldn't have had any way to know that one of their engines had not simply failed, but was straight-up gone with their wing on fire to boot.
by 0xbadcafebee | 2025-11-05 02:31:41+00:00
> The biggest threat to cloud vendors is that everyone wakes up tomorrow and cost optimizes the crap out of their infrastructure

Well there's no danger of that. Even with AWS telling you exactly how to save money (they have a thousand different dashboards showing you where you can save, and even support will tell you), it'll still take you months to work through all the cost optimization changes. Since it's annoying and complicated to do, most people won't do it.

Their billing really is ridiculous. We have a TAM and use a reseller, and it's virtually impossible for us to see what we actually spend each month, what with the reseller discounts, enterprise PPA, savings plans, RIs, and multiple accounts. Their support reps even built us some kinda custom BI tool just to look at costs, and it's still not right.
by clipsy | 2025-11-05 02:31:53+00:00
> noticed Sheridan's name as an actor or director or writer or producer on seemingly damn near every major US TV and movie production in the past five to ten years.

He's credited on 9 movies total with a tenth to be released, and none of them particularly high profile. As far as TV, he is pretty prolific recently but the only way you could possibly think he's involved in "damn near every" TV production in recent years is if you exclusively watch westerns.

(Credit where credit's due: I just learned he wrote Sicario, which was an excellent movie imo.)

Edit - Citation: https://en.wikipedia.org/wiki/Taylor_Sheridan#Filmography
by socketcluster | 2025-11-05 02:33:18+00:00
I co-founded a holiday real estate-backed crypto project in 2019. We only raised $50k though. Just enough to buy 1 unit in South Africa. We started using profits to do on-chain token buybacks with proof of burn. We proved the concept but scale was too small with just one apartment. It was a really cool concept though. Too bad the powers-that-be didn't select us.

The main problem with blockchain projects is that they're not decentralized in the way that counts. You still have to be chosen by the powers-that-be. There's still only one path to success; just like the rest of the tech sector.
by eru | 2025-11-05 02:33:37+00:00
If you just want to explore a functional language from the ML family to get some familiarity, Haskell might be the better bet.

If you specifically want a strict language and explore the OCaml ecosystem (or you already know Haskell): have fun with OCaml!

I've used both extensively---in addition to Racket, Erlang, Clean, etc, and various imperative languages.
by exidy | 2025-11-05 02:33:43+00:00
It's a risk mitigation measure. The three or six month requirement usually comes from how long you could stay, not how long you do stay.

You are also assuming a point-to-point trip. While a citizen of a country usually cannot be denied entry to their own country, any countries you transit are under no obligation to allow you through on an expired passport.

tl;dr filters out people who may be problematic to deport.
by mattstainton001 | 2025-11-05 02:34:23+00:00
True, but I think you're touching on something important regarding value. Value is different depending on the consumer: for you, you're willing and able to manage more of the infrastructure than someone who has a more narrow skillset. Being able to move the responsibility for areas of the service on to the provider is what we're paying for, and for some, paying more money to offload more of the responsibility actually results in more value for the organization/consumer
by frez1 | 2025-11-05 02:34:35+00:00
It sounds like the author and their team were more comfortable with node.js than python. they acknowledge fastapi was a good alternative that could solve their issues and allow some code reuse, but decided not to because they just wanted to use node.

the gist of this blog post is this company knew and understood node better than python, so they migrated to what they knew.
by toomuchtodo | 2025-11-05 02:35:13+00:00
Coal decline citations:

https://cleantechnica.com/2025/11/03/us-coal-exports-drop-11...

https://cleantechnica.com/2025/04/20/chinas-coal-generation-...

https://www.eia.gov/todayinenergy/detail.php?id=66524

https://blogs.worldbank.org/en/opendata/weakening-demand--st...
by theandrewbailey | 2025-11-05 02:36:11+00:00
WebP isn't enough of an improvement to justify the overhead of supporting it for me. AVIF is twice as efficient as JPEG, and has 97+% support across browsers. Use <picture> for JPEG fallback.

https://caniuse.com/?search=avif

https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/...
by epolanski | 2025-11-05 02:36:49+00:00
Then deliver the news and have constructive discussions.

For what it's worth is well accepted among most fascism historians that racism, at least in the sense of adopting racial laws and such came late, and mostly as a byproduct of the German alliance.

As for what did Italy do itself before that, if you're referring to the wars in Africa, that has nothing to do with fascism, and the two biggest colonial powers at the time were both very sane democracies.
by eru | 2025-11-05 02:36:50+00:00
Do you want to throw out out-of-order-execution and pipelining while you are at it, too?

I'm semi-serious: there are actually modern processor designs that put this burden on the programmer (or rather their fancy compiler / code generator) in order to keep the silicon simple. See eg https://en.wikipedia.org/wiki/Groq#Language_Processing_Unit
by Jtsummers | 2025-11-05 02:37:57+00:00
> The crew wouldn't have had any way to know that one of their engines had not simply failed, but was straight-up gone with their wing on fire to boot.

I don't know about the MD-11 itself, but other aircraft from that time period have sensors to detect and report overheat and fire in various parts of the aircraft, including engines and wings.
by MassPikeMike | 2025-11-05 02:39:00+00:00
I am usually working with historical documents, where both Otsu and adaptive thresholding are frustratingly almost but not quite good enough. My go-to approach lately is "DeepOtsu" [1]. I like that it combines the best of both the traditional and deep learning worlds: a deep neural net enhances the image such that Otsu thresholding is likely to work well.

[1] https://arxiv.org/abs/1901.06081
by alephnerd | 2025-11-05 02:39:18+00:00
The maintenance and inspection tends to be done by MROs, and any institutional issue within UPS's MRO would have been identified before the shutdown by the FAA and other regulators.

But based on your comment history, you aren't from the US, have not ever visited America, do not care to visit America, and haven't interacted with Americans, so I doubt you have on the ground experience with the US. But that also leads to the question of why you even care to comment on our affairs if you dislike us to such a degree.
by jongjong | 2025-11-05 02:39:33+00:00
Yes it's some weird faux-communism with all of the problems of communism and none of its benefits.

At least communism must pretend to serve the common man.

There's a point where I'd rather live in a system which at least pretends to serve people... Then at least we'd get some half-assed semblance of fairness and the discrepancies would be a subject of conversation...

When you live in a system and adhere to its rules, it's nice to be getting something of approximately equal value in return.
by leptons | 2025-11-05 02:39:45+00:00
Way back in the 90's I used to date a pretty crazy Christian girl who saw my tech prowess as useful for fighting "the mark of the beast" which she thought was some kind of future government electronic ID. While she was kind of nuts about Christian conspiracies, I actually kind of agreed about where we were headed with electronic surveillance - even back in the 90's we kind of saw this coming.
by avalys | 2025-11-05 02:39:45+00:00
Of course it was not unpreventable, though it might turn out that preventing it would have been unreasonably expensive.

But, the FAA inspectors are not responsible for making sure planes are safe to fly. They are responsible for making sure the people whose job that actually is, are doing their jobs effectively. That’s a critical difference.

It’s UPS maintenance personnel who are responsible for making sure that UPS planes are safe to fly. Yes, it’s possible that there is some institutional failure at UPS, that could have been caught if FAA inspectors were working in the past 30 days, but this isn’t the most likely scenario, and the root cause and responsibility (in this hypothetical) would still lie with UPS and not the FAA and the shutdown.