by majormajor | 2025-08-07 01:55:47+00:00
> As such, the current Coding Agent paradigm seems very well-suited for senior engineers. A lot of the skillsets are the same, only instead of having to persuade other teams you just write a deisgn doc and fire off a bunch of agents, review their work, and if you don't like their outputs, you can try again or drop down to manual coding.

I have tried this a few times, it's not there yet. The failures are consistently-shaped enough to make we wonder about the whole LLM approach.

Compared to handing off to other engineers there are a few problems:

- other engineers learn the codebase much better over time, vs relying on either a third party figuring out the right magic sauce to make it understand/memoize/context-ize your codebase or a bunch of obnoxious prompt engineering

- other engineers take feedback and don't make the same types of mistakes over and over. I've had limited luck with things like "rules" for more complex types of screwups - e.g. "don't hack a solution for one particular edge case three-levels deep in a six-level call tree, find a better abstraction to hoist out the issue and leave the codebase better than you found it"

- while LLMs are great at writing exhaustive coverage tests of simple functionality, they aren't users of the project and generally struggle to really get into that mindset to anticipate cross-cutting interactions that need to be tested; instead you get a bunch of local maxima "this set of hacks passes all the current testing" candidate solutions

- the "review" process starts to become silly and demoralizing when your coworker is debating with you about code neither of you wrote in a PR (I sure hope you're still requiring a second set of human eyes on things, anyway!)

If you have a huge backlog of trivial simple small-context bugs, go nuts! It'll help you blow through that faster! But be prepared to do a lot of QA ;)

Generally I'd call most of the issues "context rot" in that even after all the RL that's been done on these things to deal better with out-of-distribution scenarios, they still struggle with the huge amount of external context that is necessary for good engineering decision making in a large established codebase. And throwing more snippets, more tickets, more previous PRs, etc, at it seems to rapidly hit a point of diminishing returns as far as its "judgement" in picking and following the right bits from that pile at the right time.

It's like being a senior engineer with a team of interns but who aren't progressing so you're stuck as a senior engineer cleaning up crappy PRs constantly without being able to grow into the role of an architect who's mentored and is guiding a bunch of other staff and senior engineers who themselves are doing more of the nitty gritty.

Maybe the models get better, maybe they don't. But for now, I find it's best to go for the escape hatch quickly once things start going sideways. Because me getting better at using today's models won't cause any generational leap forward. That feels like it will only come from lower level model advances, and so I don't want to get better at herding interns-who-can't-learn-from-me. Better for now to stick to mentoring the other people instead.