MurphyA/hacker-news huggingface dataset:
Noob questions about Data Engineering
by mrkaka | 2025-09-13 12:42:20+00:00
Greetings! Im currently studying Data Engineering in a 2 year course/trade school and wanted to seek some insight from people that know more than me about this industry.

Which newer tools/frameworks/(maybe other fields?)are actually worth learning early?

Is there a risk of the data engineer role being "absorbed" into ML engineering or do you see it growing as a distinct path?

What is your favorite/least favorite part of the job? / What do you find difficult about it day to day personally?

Any books you recommend? Just finished reading designing data intensive applications by martin kleppmann a month ago. Also just started looking at "Fundamentals of Data Engineering" by Joe Reis & Matt Housley.

I apologize for some of these "cliche" questions that probably have already been asked, Im really just seeking mentorship from someone with any experience in the job, any advice is very appreciated!

Thank you!
by bayindirh | 2025-09-13 12:42:36+00:00
> My gripe was actually only about the post, not about the blog.

Hey, no worries. I replied to you in the context of the post, not the blog, actually. The blog has some other interesting features, too, but your comment was very clear that you wanted a TL;DR: on the post itself. :)

> You can down vote me to ground.

As a matter of principle, I don't downvote people. On the other hand, you can't downvote direct replies anyway, but that's irrelevant to my stance.

> But you need to realize that this is HN, the feedback garden.

We're both here for ~8 years, so I believe we're both pretty proficient about how this place works.

> And it did not happen to be that way. It is intentional.

Yes.

> Also, your blog didn't just "happen" to be public. You posted about it on HN and you are curious about the HN comments on your post... [snipped for brevity].

Yes and no. I post my new blog posts to two places. Here, and a Discord server which I frequent. I get almost no feedback from either. If people read and like what they read, it's great. If not, I don't care. Both blog and the digital garden is written like nobody's gonna read it. There's no rush, no anxiety, no optimization. I cook an idea, and refine it until I'm happy, and post it when I feel it's ready to be there. That blog is a public chronicle of my thoughts and what I go through.

> It is like looking into mirror and asking it not to show things you don't like. You need to come to accept the feedback, since you asked for it.

I'm pretty content with any comment incl. yours. I'm not someone who morphs to please people. Comments which challenge my words are invaluable as long as they don't become ad-hominem attacks. It both helps me to see my dark corners and improve my discussion and language skills (this is not my native language, tbh).
by skydhash | 2025-09-13 12:42:42+00:00
> Add logging throughout some code

That's one of the thing that I wouldn't delegate to LLM. Logging is like a report of things that happens. And just like a report, I need relevant information and the most useful information.

...

A lot of these use cases actually describes the what. But the most important questions is always the why. Why is it important to you? Or to the user? That's when things have a purpose and not be just toys.
by dredmorbius | 2025-09-13 12:42:55+00:00
... if I am listening to music in YouTube it should be trivial to find the browser tab involved ...

A key reason why I tend to access media through a media player (usually mpv in a terminal, occasionally others), and would favour a Web model which divides textual content, media, retail/commerce, and apps into their own apps.

That is, not an app per retail site, but a retail app which manages payment, reputation, identity, and related tasks. Shoehorning everything into "the Web browser" is a category error IMO.
by mike_hearn | 2025-09-13 12:43:30+00:00
Every modern commercial OS is a hybrid architecture these days. Generally subsystems move out of the kernel when performance testing shows the cost isn't too high and there's time/money to do so. Very little moves back in, but it does happen sometimes (e.g. kernel TLS acceleration).

There's not much to say about it because there's never been an actual disagreement in philosophy. Every OS designer knows it's better for stability and development velocity to have code run in userspace and they always did. The word microkernel came from academia, a place where you can get papers published by finding an idea, giving it a name and then taking it to an extreme. So most microkernels trace their lineage back to Mach or similar, but the core ideas of using "servers" linked by some decent RPC system can be found in most every OS. It's only a question of how far you push the concept.

As hardware got faster, one of the ways OS designers used it was to move code out of the kernel. In the 90s Microsoft obtained competitive advantage by having the GUI system run in the kernel, eventually they moved it out into a userland server. Apple nowadays has a lot of filing systems run in userspace but not the core APFS that's used for most stuff, which is still in-kernel. Android moved a lot of stuff out of the kernel with time too. It has to be taken on a case by case basis.
by n4r9 | 2025-09-13 12:43:31+00:00
The ten minutes I watched was one big appeal to authority. "Here's Doctor X, a highly respected scientist. Look at all the big name universities he's linked to. Listen to him waffle on about how he thinks climate science is corrupt."

Appeal to authority is a dubious fallacy to invoke in the first place. If I need to assume something about (let's say) geology - which I know little about and haven't the time to research myself - I'm going to trust the general consensus of professional geologists. I'm not going to waste my time listening to a marine biologist who sounds like a crank and claims they've discovered that the whole field is bogus.

"If you're interested in finding the truth, you'll at least" check out this (surprise surprise, textual) take-down of the movie, with a comprehensive set of links debunking (in, surprise surprise, text form) the hackneyed climate myths that it brings up: https://skepticalscience.com/climate-the-movie-a-hot-mess-of...
by agentcoops | 2025-09-13 12:43:37+00:00
I have a similar relation to AI with programming -- and my sense is very many HN readers do as well, evidenced not least by the terrific experience report from antirez [1]. Yet it is rare to see such honest and open statements even here. Instead, HN is full of endless anti-AI submissions on the front page where the discussion underneath is just an echo chamber of ill-substantiated attacks on 'AI hype' and where anything else is down-voted.

It's what is, to me, so bizarre about the present moment: certainly investment is exceptionally high in AI (and of course use), but the dominant position in the media is precisely such a strange 'anti-AI hype' that positions itself as a brave minority position. Obviously, OpenAI/Altman have made some unfortunate statements in self-promotion, but otherwise I genuinely can't think of something I've read that expresses the position attacked by the anti-AI-ers -- even talk of 'AGI' etc comes from the AI-critical camp.

In a sense, the world seems divided into three: obvious self-promotion from AI companies that nobody takes seriously, ever-increasingly fervent 'AI critique', and the people who, mostly silent, have found modern AI with all its warts to be an incomparably useful tool across various dimensions of their life and work. I hope the third camp becomes more vocal so that open conversations about the ways people have found AI to be useful or not can be the norm not the exception.

[1] https://antirez.com/news/154
by sdfssdf | 2025-09-13 12:45:08+00:00
A look at destatis tells me something else for Germany (in 2024): Solar has a share of 15 %, and wind 28 %. In total 57 % of the produced energy comes from renewable sources. (https://www.destatis.de/DE/Themen/Branchen-Unternehmen/Energ...)
by jfengel | 2025-09-13 12:45:40+00:00
Email is still completely open. Even Usenet still exists. There may be more people on it now than there were in the 80s, just because it was so tiny then. (The entirety of Usenet before Eternal September fits on a thumb drive.)

I believe that what has changed is less about technology or even money, but about people. In your time frame, everyone on the Internet was an academic techie. You could bump into a random person on IRC and have something to talk about.

You can connect with vastly more people today, but they are less likely to be of interest to you. You're spoiled for choice: there are now a trillion chat rooms instead of a thousand. It's harder to find your people.
by darkpicnic | 2025-09-13 12:46:18+00:00
I created an open source tool to help apps stay near the Dunbar Number: https://highlyprobable.io/articles/ten-cubed. I think the concept of social networks is interesting, but the ultimate unbounded result is a disaster.

Edit (missing link to github repo): https://github.com/darkpicnic/ten_cubed
by ghuntley | 2025-09-13 12:47:17+00:00
This is the 9-month recap of my "The Future Belongs to People Who Do Things" talk.

Inside:

- The problems with AGENTS . md

- The problems with LLM model selectors

- Best practices for LLM context windows

- AI usage mandates at employers

- Employment performance review dynamic changes

- The world's first vibe-coded emoji RPN calculator in COBOL

- The world's first vibe-coded genz compiler (CURSED)

and a final urge to do things, as this is perhaps the last time I deliver this talk. It's been nine months since the invention of tool-calling LLMs, and VC subsidies have already started to disappear.

If people haven't taken action, they're falling behind because it's becoming increasingly cost-prohibitive to undertake personal upskilling.
by ChrisMarshallNY | 2025-09-13 12:48:53+00:00
I have a similar view of AI.

I find it best as a "personal assistant," that I can use to give me information -sometimes, highly focused- at a moment's notice.

> The potential for then stunting the growth of actual juniors into tomorrow's senior developers is a serious concern

I think it's a very real problem. I am watching young folks being frozen out of the industry, at the very beginning of their careers. It is pretty awful.

I suspect that the executives know that AI isn't yet ready to replace senior-levels, but they are confident that it will, soon, so they aren't concerned that there aren't any more seniors being crafted from youngsters.

Would suck, if they bet wrong, though…
by codethief | 2025-09-13 12:50:07+00:00
> Whereas I believe a right to privacy in your tools and communication is essential, he believes they infringe on the government's ability to catch criminals.

Honestly, if I had spent 30 years of my life trying to catch criminals, I would probably believe the same. Just like dermatologists telling everyone to put on sunscreen at all times (because they see skin cancer cases on the daily), criminal prosecutors often live in a bubble, where crime happens all the time and everywhere, and you can never have enough sophisticated tools to catch the perpetrators. They completely forget that normal life away from crime exists, too.
by ToucanLoucan | 2025-09-13 12:50:18+00:00
Their entire business model? Making search results worse to juice page impressions? Every dark pattern they use to juice subscriptions like every other SaaS company? Brand lock-in for Android? Paying Apple for prominent placement of their search engine in iOS? Anti-competitive practices in the Play store? Taking a massive cut of Play Store revenue from people actually making software?
by swannodette | 2025-09-13 12:51:14+00:00
Yes! Recent personal hacks with advice interactively programmed with the help of Claude 4.1 chat session running in Emacs.

- popping the global mark to travel backwards to previous edit points would reuse the same Emacs window instead of using a window that is already showing the buffer. Fixed - I would accidentally destroy my window configuration w/ C-x 1. Fixed, use advice to automatically save the window configuration into a register if I invoke the command to remove all other windows. Now I can easily recover if I make that mistake. - I want to be able to select and then scroll any other open window w/o leaving the current one. Fixed - A crazy one. I collect note w/ links or whatever for reading/watching later, these are marked w/ a timestamp. My notes file is not an agenda file, i.e. not filled with todos / tasks. I made a hack to temporarily include the current non-agenda org-mode buffer in the agenda list and then show inactive timestamps. Now I can scan a day/week/month for interesting notes I took. This doesn't interact at all with my real agenda. - org-agenda opens items in weird places, use advice to fix it so that it always appears where I like. - fix inf-clojure so that it uses dep.edn as project root over .git

I used IntelliJ happily for 10 years (I was a heavy Emacs user for 10 years before that). While it's true that some things are a little less convenient (I don't use LSP), knowing that I can tailor things exactly to my tastes is a serious breath of fresh air.

It used to be I mostly used IntelliJ for work/OSS and Emacs for org-mode. Now the situation is likely reversed. Emacs for work/OSS and IntelliJ only if I need step debugging/global refactoring.
by HPsquared | 2025-09-13 12:51:58+00:00
See also all the hype about rare earths being needed for electric motors and generators, or cobalt for batteries. Neither of those are really strictly needed. Induction motors can be used instead which don't require any magnets (but those are a few percent less efficient); Lithium Iron Phosphate batteries don't need cobalt (they are slightly less powerful but cheaper and safer).
Ask HN: Is it worth going to Germany to do my bachelors in CS/AI?
by DingDongPing | 2025-09-13 12:52:49+00:00
I live in India and done with my first year of uni(pretty decent uni) but I don't like it here as its in the same city i live and I wanna venture out. USA is pretty expensive and more importantly I wont get a good uni there because of my marks but I indeed plan to move to or at least want to move to silicon valley, what I am thinking is to do bachelors in CS/AI in Germany then work there for a few years to earn enough and then go to the US for masters. Is it a good idea and more importantly is it feasible?
by ericmay | 2025-09-13 12:52:54+00:00
Japan also has less social mobility - it’s a great country but it’s no paradise.

Homes get torn down all the time because they aren’t worth anything - not exactly environmentally friendly.

Even in America today you have plenty of things worth investing in that don’t have to be homes if you can’t stomach the initial investment. Idk what you envision about replacing investment in homes with “research” though certainly curious to learn more about what you envision.
by growse | 2025-09-13 12:54:07+00:00
> But by kumbaya ethics or community spirit or whatever you want to call it, they're scummy vultures.

> They're dipping into the "take a penny" jar and not replenishing it.

> I can't square this with any sense of justice or morality.

Speak for yourself. Other people have different values and therefore interpretations of what's "moral" or "just".

If you want to make the argument that the only use of open source that's moral is one in which the user contributes back, that's fine. But I think you'll find yourself in a minority.
by FirmwareBurner | 2025-09-13 12:54:23+00:00
>There's no other way to have a true democracy than to make things as equal as possible.

Only if by that you mean equal opportunities for everyone.

But if you mean equal outcomes, then you're guaranteed to get USSR/Cuba/Venezuela poverty, famines and shortages, and even there that didn't fix the issue of the elites being super wealthy, it just made everyone else equally poor.

People will never end up equal no matter how many thumbs the government puts on the scale, that actually makes it so much worse.
by smokel | 2025-09-13 12:55:19+00:00
Google Gemini 2.5 Pro actually has a quite nuanced reply when asked to consider this statement, including the following:

> "Massive privacy invasion: The core of modern adtech runs on tracking your behavior across different websites and apps. It collects vast amounts of personal data to build a detailed profile about your interests, habits, location, and more, often without your full understanding or consent."
by Krasnol | 2025-09-13 12:55:38+00:00
This is the actual judgement: https://eur-lex.europa.eu/legal-content/en/TXT/?uri=CELEX:62...

One of the reasons they judged this way is the lack of feasible low-carbon alternatives. Which is hilarious because we're talking about EU taxpayer money here. Money that can be spend only once. If you spend it on a nuclear reactor which may or may not be build within 10 years, you can't spend it on those "feasible low-carbon alternatives" which we already have and which would have produced a lot of energy in that construction time.

The court however said also, that the strict regulations, which are often the main argument of nuclear fans, are not to be lowered. Therefore there is no outcome for nuclear where it will get cheaper. The French, which have never stopped building nuclear all over the world, still didn't manage to get it cheaper. There are always cost explosions which cause the energy price to rise over time.

Which brings us back to why we're even talking about this:

France has a fleet which needs more and more maintenance. It is costly and the state already supports every Watt of nuclear energy through their tarif bleu. They desperately need this EU money to support this show. Without it, they'd be forced to innovate and expand on renewables, like the rest of the planet.

...but just like the Germans with their fossil fuel cars, they'll try to keep it alive as long as possible, even though the market comes apart around them.
by m101 | 2025-09-13 12:56:02+00:00
Sure, if someone wishes to completely disconnect from the grid then they are free to do so. I don't think there is anyone stopping anyone who chooses to do this.

The problem arises when someone wishes to remain connected to the grid so that the grid supports them over the winter/over night/when there's bad weather/when their batteries run out etc etc.

One has to pay for the costs of providing you with power all the time, not just when your solar panels aren't working.

Now the question of why it is economically advantageous to self generate these days over the economies of scale of the normal power industry is a really interesting question. I think the reason for this is ultimately: it is because of a total failure of governments and the energy businesses to provide cheap energy. Something which we know to be possible, but they have failed. I put the blame with governments personally.
by insane_dreamer | 2025-09-13 12:56:25+00:00
I understand and sympathize with the interest in keeping American jobs for Americans first, if there aren't enough to go around. Generally makes sense. I feel the same way about my kids chances of getting into a good state college (increasingly difficult when you're competing against kids from the whole world).

But it's the height of stupidity to employ ICE "thugs" to hunt down and round up poor laborers doing jobs that most Americans don't want to do, while letting big companies hire lots of foreigners on H1Bs for SWE jobs, while at the same time you have Americans graduating from college and unable to find jobs.

The US should get rid of ICE and drop the H1B program altogether -- (maybe with some narrow exceptions and not even sure about that). For exceptionally talented people wanting to work in the US there's the EB1 and EB2 programs. That would both largely solve the "illegals are taking our jobs!" problem and stop us acting like some 3rd world police state with masked police acting like the Stasi.
by ddahlen | 2025-09-13 12:56:29+00:00
One good reason to keep the raw data and orientation separate is hardware calibrations. I see some discussion here along the lines of "why not just flip the data before saving it", and one counter-argument is that you should store the flip metadata anyways, as it important to trace which hardware pixel corresponds to which stored pixel. I realize this information is not vital for everyone, but it is super useful in fields where you need to characterize the hardware performance at the pixel level (for example astronomy).
by athrowaway3z | 2025-09-13 12:56:30+00:00
Both CRUD and boilerplate are arguably a tooling issue. But there are also a bunch of things only AI will let you do.

My tests with full trace level logging enabled can get very verbose. It takes serious time for a human to parse where in the 100 lines of text the relevant part is.

Just telling an AI: "Run the tests and identify the root cause" works well enough, that nowadays it is always my first step.
by imiric | 2025-09-13 12:56:45+00:00
That's amazing! It would be very interesting to see what a config spanning 35 years looks like, and what your workflow is. If you could write a blog post about it, I'm sure others would appreciate it as well.

I suppose this goes for any Emacs user. There's so much we can learn and be inspired from by seeing how other people use it.
by 8cvor6j844qw_d6 | 2025-09-13 12:58:08+00:00
Just started using Claude Code recently.

It seems to speed up feature development but requires one to have a good understanding of the codebase to guide it and be aware of edge cases it missed.

Also, it doesn't seem to be able to take advantage of latest information or new SDK features unless deliberately informed. Not sure if I'm doing it right, but I resorted to feeding it documentation when it can't seem to something right.

The only thing I'm still unsure is the context management with /compact and /clear
by teamonkey | 2025-09-13 12:58:28+00:00
> I think a good exercise for the reader is to reflect on why they were ever against nuclear power in the first place.

The context is a long string of nuclear incidents throughout the Cold War through to the ‘90s.

Not just Chernobyl, not just Fukushima, but the string of disasters at Windscale / Sellafield and many others across the globe.

https://en.wikipedia.org/wiki/List_of_nuclear_power_accident...

https://en.wikipedia.org/wiki/List_of_nuclear_and_radiation_...

These disasters were huge, newsworthy and alarmingly regular. People read about those getting sick and dying directly as a result. They felt the cleanup costs as taxpayers. They saw how land became unusable after a large event, and, especially terrifying for those who had lived as adults through Cold War, saw the radioactive fallout blown across international borders by the wind.

It’s not Greenpeace or an anti-nuclear lobby who caused the widespread public reaction to nuclear. It was the public reaction seeing it with their own eyes, and making an understandable decision that they didn’t like the risks.

Chernobyl was one hammer blow to the coffin lid, Fukushima the second, but nuclear power was already half-dead before either of those events, kept alive only by unpopular political necessity.

I’m not even anti-nuclear myself, but let’s be clear: the worldwide nuclear energy industry is itself to blame for the lack of faith in nuclear energy.
by anonymars | 2025-09-13 13:01:27+00:00
I guess different people's experiences are different. Seems needlessly snarky. I have no memory of this yet can rattle off (among others)

- Hindenburg

- Pan Am 103

- Tenerife

- TWA 800

- JAL 123

- Korean 007

- Iran Air shootdown

- MH 17

- MH 370

- The DC-10 cargo door (Turkish airlines)

- The DC-10 fan disk (United airlines)

- The Hawaiian Airlines 737 fatigue decompression

- Gimli glider

- Miracle on the Hudson

- September 11