by rhdunn | 2025-04-06 10:53:14+00:00
Part of the problem is that it is difficult/impossible to go through the original sources and verify the claims. Another part of the problem is knowing what biases an outlet/reporter has and what parts of a story/the thing they are reporting on is missing out.

It's known that a few research papers (e.g. [1], [2]) have been fabricated on various topics. So when you read the news about those, do you read the papers? If you do, can you verify the claims in those papers? It's not possible to do at scale, so you have to trust the reporting on those stories.

A lot of reporting on things like that, or releases from major companies, tend to be based on press releases. You also have papers and websites that use others as their source of the information, so any inaccuracies or biases in that initial reporting gets magnified. It also depends on the knowledge of the person reporting -- if they are not an expert in the field they can easily get some of the details wrong.

It took a long time, combined effort, and a lot of resources for the Leela Zero team to replicate the results of Alpha Zero [3]. And that's with papers that don't have the source code, or exact details on the training methodology. I've seen quite a lot of papers on things like arXiv that amount to a summary of "we did a thing, here's a table of results" -- this makes it hard to replicate the results. And a lot of papers build on work of others, so quite often you'll need to read and understand hundreds of papers.

Then you have reporting from trials. They will not cover everything in the trial, but provide summaries of the results. See both the Lucy Letby case and the Johnny Depp vs Amber Herd trails for how those are reflected in the relevant media, both during and after the trials. You can't go to all the trials and even when there are recodings of them there will be multiple weeks of video to go through, which is impossible for all trials. So you have to rely on reporting or reports from the people who have done that, and are reliant on the witnesses and evidence given during the trial to be accurate and in support of the outcome of that trial.

Then there's reporting of things like campaign events and rallies during elections. Media will only show soundbites that can remove the context from the original -- see also the countless "out of context" videos on YouTube. You can't always get the original sources, an you can't always go through the entire context. So you have to rely on the media reporting. That gives wildly different perspectives from the different groups on a given topic -- see e.g. the left and right reporting/coverage on things like Donald Trump doing the stint at a McDonalds.

I don't have any answers on how best to solve this other than:

1. listing (and preserving) sources reported on that are not anonymous and can be verified -- including linking back to any press releases, research papers, trail transcripts, videos, or other relevant sources;

2. having a mechanism for replicating results from research papers, and having a way to find if/how that was done for a given research paper;

3. consuming reports from different sources, ideally ones that have different perspectives.

[1] https://www.science.org/content/article/researchers-plan-ret...

[2] https://www.alzheimers.org.uk/for-researchers/explaining-amy...

[3] https://lczero.org/blog/